diff --git a/fep/1311/cow.jpg b/fep/1311/cow.jpg new file mode 100644 index 0000000..2680c5e Binary files /dev/null and b/fep/1311/cow.jpg differ diff --git a/fep/1311/fep-1311.md b/fep/1311/fep-1311.md new file mode 100644 index 0000000..830dd80 --- /dev/null +++ b/fep/1311/fep-1311.md @@ -0,0 +1,465 @@ +--- +slug: "1311" +authors: Helge <@helge@mymath.rocks> +status: DRAFT +dateReceived: 2024-12-08 +discussionsTo: https://socialhub.activitypub.rocks/t/fep-1311-media-attachments/4796 +trackingIssue: +--- +# FEP-1311: Media Attachments + +## Summary + +Media Attachments are ubiquitous in the Fediverse. My quick investigation +into the explore tab on `mastodon.social` yields that about half the +posts contain an image attachment. The mechanism for these is poorly +documented. For example, it is not mentioned in [ActivityPub][]. + +My goal in this FEP is to document current usage, and issue recommendations +on how to improve it. These recommendations are based on the +support table _Recommended Media Attachment Format_ +available at [FunFedi.dev][FunFedi]. + +For developers that enjoy making their keyboards smoke, I believe +that the above link combined with the content of [Testing](#testing) +should be enough to adapt their Fediverse applications. +The other parts are meant for people, who which to improve +the situation related to media attachments. + +## Basic format + +We will discuss our basic suggested format with the following +example of an image attachment. + +```json +{ + "type": "Image", + "name": "A beautiful cow", + "url": "http://pasture-one-actor/assets/cow.jpg", + "width": 100, + "height": 162, + "mediaType": "image/jpeg", + "digestMultibase": "zQmaeDPzhNL32WQZnnzB1H6QJWvvFNEHdViDB71yrxyXU1t", + "size": 9045 +} +``` + +There is a lot to say here, first how does this relate to communication +in [ActivityPub][] which is done by activity. For this consider the +activity (taken from [data.funfedi.dev][data1]) given by + +```json +{ + "@context": [ + "https://www.w3.org/ns/activitystreams", + "https://www.w3.org/ns/credentials/v2", + { + "size": "https://joinpeertube.org/ns#size" + } + ], + "type": "Create", + "actor": "http://pasture-one-actor/actor", + "to": [ + "http://akkoma/users/witch", + "https://www.w3.org/ns/activitystreams#Public" + ], + "id": "http://pasture-one-actor/actor/S5Szzuugy50", + "published": "2024-12-05T08:18:48Z", + "object": { + "type": "Note", + "attributedTo": "http://pasture-one-actor/actor", + "to": [ + "https://www.w3.org/ns/activitystreams#Public", + "http://akkoma/users/witch" + ], + "id": "http://pasture-one-actor/actor/qDqgbPpNQPw", + "published": "2024-12-05T08:18:48Z", + "content": "Recommended Image Format", + "attachment": [ + { + "type": "Image", + "name": "A beautiful cow", + "url": "http://pasture-one-actor/assets/cow.jpg", + "width": 100, + "height": 162, + "mediaType": "image/jpeg", + "digestMultibase": "zQmaeDPzhNL32WQZnnzB1H6QJWvvFNEHdViDB71yrxyXU1t", + "size": 9045 + } + ] + } +} +``` + +We note that the media attachment is contained in the array of +`attachment` of the `Note` object. Furthermore, in difference +to the activity and the object, there is no `id` property +nor `actor` or `attributedTo` property. This is on purpose, +as those are inherited from the object the media attachment +is attached to. See [Content Licensing](#content-licensing) +for discussion about not having an `attributedTo` property. + +In particular, one should emphasize that a media attachment +not having an `id` is useful to signify that it is not useful +as an object without the note, it is attached to. + +### Specifying basic properties + +To use media attachments, an object MUST have an `attachment` +property, whose value is an array containing objects. +Furthermore, the contained objects MUST have a type +property. For it to be a media attachment the type +property MUST be `Audio`, `Image`, or `Video`. However, +the attachment property MAY contain other form +of attachments, e.g. [FEP-0ea0: Payment Links][0ea0]. + +This can be expressed as the json-schema: + +```json +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "attachment": { + "type": "array", + "items": { + "type": "object", + "properties": { + "type": { + "type": "string", + "examples": ["Audio", "Image", "Video"] + } + }, + "required": ["type"] + } + } + }, + "required": ["attachment"] +} +``` + +We now discuss the specific form a media attachment. +In addition to `type`, a media attachment MUST also +contain an `url` property providing the link +to the media. The `url` property MAY also be +an array, see [Multiple Media Versions](#multiple-media-versions). +However, this just to be future proof. + +Furthermore, media attachments SHOULD contain +a `name` property providing an alternative plain text +description of the media object. + +Again this can be represented as a json-schema. + +```json +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "type": { + "type": "string", + "enum": ["Audio", "Image", "Video"] + }, + "name": { + "type": "string", + "examples": ["A beautiful cow"] + }, + "url": { + "type": ["string", "array"], + "examples": ["http://you.example/image.png"] + } + }, + "required": ["type", "url"] +} +``` + +### Current state of type + +Many current implementations use `Document` for media attachments. +This has the distinct disadvantage to using `Audio`, `Image`, +or `Video` that the parser needs to inspect more content than +`type` to discover what type of attachment it is. + +Let's write a bit of python to determine if something is +a media attachment based on `mediaType`, discussed later. + +```python +def is_media(attachment:dict) -> bool: + media_type = attachment.get("mediaType") + if media_type is None: + ... # handle error case + + main_type, _ = media_type.split("/", 1) + + return main_type in ["audio", "image", "video"] +``` + +This already is fairly complicated, but there is more. If +one allows `url` to be an array, one needs a different check, +so it turns into something like: + +```python +def is_media(attachment:dict) -> bool: + url = attachment.get("url") + if isinstance(url, list): + return is_media(url[0]) + + media_type = attachment.get("mediaType") + if media_type is None: + ... # handle error case + + main_type, _ = media_type.split("/", 1) + + return main_type in ["audio", "image", "video"] +``` + +There are more exceptions and poor configuration to treat, +e.g. url could be an empty list, or the implementation +could attach `mediaType` to the full object instead +of the `Link`. + +The goal of a specification needs to be to simplify the code +that needs to be written, so we insist on people using +the types `Audio`, `Image`, and `Video` for media +attachments. + +## Properties of the linked file + +In this section, we discuss properties related to the linked file. The linked file is retrieved either by performing a `GET` request on the value of `url` or on `href` of the `Link` objects if `url` is an array. Just to mention it, this requirement might change, once [Authentication and Authorization](#authentication-and-authorization) is dealt with. + +Let's look back at our original example of a media attachment + +```json +{ + "type": "Image", + "name": "A beautiful cow", + "url": "http://pasture-one-actor/assets/cow.jpg", + "width": 100, + "height": 162, + "mediaType": "image/jpeg", + "digestMultibase": "zQmaeDPzhNL32WQZnnzB1H6QJWvvFNEHdViDB71yrxyXU1t", + "size": 9045 +} +``` + +Here the linked file is given by a `GET` on `http://pasture-one-actor/assets/cow.jpg` and the result would be + +![A beautiful cow](./cow.jpg) + +The properties `mediaType`, `digestMultibase`, and `size` could +be valid for any attached file, even a non media one, e.g. +a text document. Let's quickly review them. [mediaType](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-mediatype) is defined +in the [ActivityStreams Vocabulary][ASC]. It describes the MIME type +and tells us important information on how to render the file. + +`digestMultibase` is defined [here](https://w3c.github.io/vc-data-integrity/#resource-integrity) as part +of [Verifiable Credential Data Integrity][VC]. The +encoding of a digest in multibase with [multihash](https://www.multiformats.io/multihash/) is somewhat different to the rest of multicodec, +because one first has a byte to indicate the format, then another one +to indicate the length. This means in particular that all +`digestMultibase` using `sha-256` will start with `zQm`, the `z` indicating +base58 encoding. Checking the digest is important to ensure __integrity__. +As media is often hosted off site using S3, this seems important. +For another usage see [Content Addressed Storage](#content-addressed-storage). + +Finally `size` being the file size in bytes is borrowed from PeerTube. +The `size` should tell us if we want to preload the media or not. + +There is something missing in the file properties: _access control_, see +the section [Authentication and Authorization](#authentication-and-authorization) +in the open questions below. + +### Specifying file properties + +The creator of a media attachment SHOULD include the +values of `mediaType`, `digestMultibase`, and `size`. The +consumer of a media attachment SHOULD ensure integrity +of the downloaded attachment based on `digestMultibase`, +i.e. check the digest. The consumer of a media attachment +SHOuLD decide based on `size` and `mediaType` the best way +to consume the attachment. + +`size` and `mediaType` become more relevant when +[multiple versions of the media attachment](#multiple-media-versions) +are provided. For example, this could mean that in one +feeds one only sees the low quality video by default. + +## Properties of an image + +We have now discussed all properties of our example document +except for `width` and `height`. These properties are only +relevant for an image and a video, but not for audio. Similarly, audio and video can have a `duration`, which images don't. Finally, Mastodon has introduced the additional properties + +- [`focalPoint`](https://docs.joinmastodon.org/spec/activitypub/#focalPoint) +- [`blurHash`](https://docs.joinmastodon.org/spec/activitypub/#blurhash) + +where at least `focalPoint` is user defined. There are a lot +of other properties one can consider for media, e.g. + +- Where was the picture taken? e.g. `location` +- What is the frame rate of the video? e.g. `fps` +- Provide an album cover for audio? + +In order to standardize these things further work is needed. + +## Multiple Media Versions + +As it is currently not supported in the Fediverse, +I will just give the basic example how to use multiple +attached Links: + +```json +{ + "type": "Video", + "name": "A beautiful cow eating", + "url": [ + { + "type": "Link", + "size": 54373, + "digest": "zQmSzK5qEe5tpjwGMhmjx9RvVoPkWhEmCwxP2s7wPMpKMoK", + "width": 256, + "height": 144, + "href": "http://pasture-one-actor/assets/cow_eating.mp4", + "mediaType": "video/mp4" + }, + { + "type": "Link", + "size": 2271723, + "digest": "zQme2X4rgWuRdmAtGGMSEbdoeRQ2NAL2VptcdRGTYDZbSKG", + "width": 1920, + "height": 1080, + "href": "http://pasture-one-actor/assets/cow_eating_hd.mp4", + "mediaType": "video/mp4" + } + ], + "duration": "PT3S" +} +``` + +As the example shows, this is useful to attach both a low quality +version (54kb) and a high quality one (2.2MB) of a video. + +We think that supporting this will open the door for richer +applications. + +## Testing + +By using json-schema, one can validate some level of correctness +of generated media attachments. Relevant schemas +are available at [Fediverse schemas][schemas] for media +attachments. They can be combined into a feature test +using Gherkin, see [Media Format][MF]. + +If you wish to validate everything, including digest, +you can use the examples provided at [FunFedi.dev][FunFedi]. + +## Open Question + +This section is essentially a todo list for the community +on stuff that should be fixed, but isn't yet. + +### Content Licensing + +The picture in the examples was created based on [this picture](https://pixabay.com/photos/highland-cow-cow-cattle-animal-8678950/) available for +free on pixabay by photographer [derekmuller][derekmuller]. +Unfortunately, the current standards to not let me attach +this information to my media object. + +One could now say that this could be solved with just +using the `attributedTo` property. Unfortunately, this +has a lot of drawbacks. For example [derekmuller][derekmuller] +is not an ActivityPub actor. Also attributing my cropped +low resolution image to him, might be something he does not appreciate. +Finally, just attributing this image is probably not enough, +one should also inform people on how it is licensed. + +See [FEP-c118][c118] and its discussions for more on the topic. + +### Authentication and Authorization + +Currently, image links must be accessible without any +form of authorization. This is due while communication +between a user and their server requiring authentication +and between servers requiring authentication, images +are often stored on third party services, e.g. S3, +thus adding authentication is hard. + +For some approaches to resolve this see [this Fediverse discussion][silverpill1]. + +One approach to achieve authentication and authorization +easily with existing technologies would be [Bearcaps][bearcaps]. + +For a different approach see also [Binary Fediverse transport](#binary-fediverse-transport). + +### Content addressed storage + +Storing media is costly. It is thus important to avoid +duplication. By having a digest for all media through +the `digestMultibase` property, we can use this to +index our media storage. This means that before downloading +a file, we can check if we already have it. + +### Mixed media content + +Consider posting a song, e.g. something from the brat summer, then +you might want to attach the album cover, e.g. an image featuring +the color `#8ACE00`. Maybe you will also want to attach some +lyrics. This means that your media content contains three parts +of separate media type. + +One might want to extend the schema for media attachments to convey +this information. + +### Binary Fediverse transport + +A failure of [ActivityPub][] is that it restricts transport to +be JSON. ActivityPub thus forces people to use external means, e.g. +download the file, to convey media content. + +One could solve some problems, e.g. +[Authentication and Authorization](#authentication-and-authorization), +by just allowing transport of binary blobs. This would require +an extension of the wire format. + +Allowing for messages to contain binary blobs would also for +sharing media via thick clients. + +## References + +- Christine Lemmer Webber, Jessica Tallon, [ActivityPub][ActivityPub], 2018 +- Helge, [FunFedi.dev: Recommended Media Attachment Format][FunFedi] +- Helge / funfedi.dev [Interoperability Data for the Fediverse][data1], akkoma v3.13.3, inputs 0.1.11, Recommended Media Attachment Format, example 1 +- Tim Bray, [FEP-c118: Content licensing support][c118] +- [Pixabay user derekmuller][derekmuller] +- silverpill, [FEP-0ea0: Payment Links][0ea0] +- Helge, [Feature: Media Attachments][MF] part of fediverse-features +- Helge, [Schemas for the Fediverse][schemas] +- Mastodon, [Bearcaps][bearcaps] + +## Posts + +- [Social Media Post][silverpill1] by silverpill et al. Relevant parts: + +> Apparently Streams has some mechanism for protecting attachments. Media URLs in non-public posts look like this: +> https://{domain}/photo/{filename}.jpg?token={token} +> +> IIRR at least in Hubzilla that token is just part of OpenWebAuth's "magic authentication". Where I guess the token contains info about which instance(s?) to contact in order to verify your identity. The audience is kept in the media server(s) database and sync'ed between clones. + +[ActivityPub]: https://www.w3.org/TR/activitypub/ +[FunFedi]: https://funfedi.dev/support_tables/generated/recommended_attachments/ +[data1]: https://data.funfedi.dev/0.1.11/akkoma__v3.13.3/recommended_attachments +[c118]: https://codeberg.org/fediverse/fep/src/branch/main/fep/c118/fep-c118.md +[derekmuller]: https://pixabay.com/users/derekmuller-1225161/ +[silverpill1]: https://mitra.social/post/01939c02-dedc-dc4c-0431-a5fe7925b337 +[0ea0]: https://codeberg.org/fediverse/fep/src/branch/main/fep/0ea0/fep-0ea0.md +[ASC]: https://www.w3.org/TR/activitystreams-vocabulary/ +[VC]: https://w3c.github.io/vc-data-integrity/ +[MF]: https://codeberg.org/helge/fediverse-features/src/branch/main/features/schema/schema_media.feature +[schemas]: https://schemas.funfedi.dev/ +[bearcaps]: https://docs.joinmastodon.org/spec/bearcaps/ + +## Copyright + +CC0 1.0 Universal (CC0 1.0) Public Domain Dedication + +To the extent possible under law, the authors of this Fediverse Enhancement Proposal have waived all copyright and related or neighboring rights to this work.