granary package
Reference documentation.
as1
Utilities for ActivityStreams 1 objects.
- granary.as1.object_type(obj)[source]
Returns the object type, or the verb if it’s an activity object.
Details: http://activitystrea.ms/specs/json/1.0/#activity-object
- granary.as1.get_object(obj, field='object')[source]
Extracts and returns a field value as an object.
If the field value is a string, returns an object with it as the id, eg
{'id': val}
. If the field value is a list, returns the first element.
- granary.as1.get_objects(obj, field='object')[source]
Extracts and returns a field’s values as objects.
If a field value is a string, generates an object with it as the id, eg
{'id': val}
.
- granary.as1.get_owner(obj)[source]
Returns an object’s author or actor.
Prefers author, then actor. If that’s an object, returns its id, if available. If neither field exists, but
obj.objectType
is inACTOR_TYPES
, returns its own id, if available.For posts, updates, and deletes, falls back to the inner object’s owner if the outer activity has no actor.
- granary.as1.get_url(obj)[source]
Returns the url field’s first text value, or
''
.Somewhat duplicates
microformats2.get_text()
.
- granary.as1.get_ids(obj, field)[source]
Extracts and returns a given field’s values as ids.
Returns string values as is. For dict values, returns their inner
id
orurl
field value, in that order of precedence.
- granary.as1.merge_by_id(obj, field, new)[source]
Merges new items by id into a field in an existing AS1 object, in place.
Merges new items by id into the given field. If it exists, it must be a list. Requires all existing and new items in the field to have ids.
- granary.as1.is_public(obj)[source]
Returns True if the object is public, False if private, None if unknown.
…according to the Audience Targeting extension: http://activitystrea.ms/specs/json/targeting/1.0/
Expects values generated by this library:
objectType
group
,alias
@public
,@unlisted
, or@private
.Also, important point: this defaults to True, ie public. Bridgy depends on that and prunes the to field from stored activities in Response objects (in
bridgy/util.prune_activity
). If the default here ever changes, be sure to update Bridgy’s code.
- granary.as1.add_rsvps_to_event(event, rsvps)[source]
Adds RSVP objects to an event’s attending fields, in place.
- granary.as1.get_rsvps_from_event(event)[source]
Returns RSVP objects for an event’s attending fields.
- granary.as1.activity_changed(before, after, log=False)[source]
Returns whether two activities or objects differ meaningfully.
Only compares a few fields:
objectType
,verb
,content
,location
, andimage
. Notably does not compareauthor
,published
, orupdated
.- Parameters:
before – dict, ActivityStreams activity or object
after – dict, ActivityStreams activity or object
- Return type:
- granary.as1.append_in_reply_to(before, after)[source]
Appends before’s
inReplyTo
toafter
, in place.
- granary.as1.actor_name(actor)[source]
Returns the given actor’s name if available, otherwise Unknown.
- granary.as1.original_post_discovery(activity, domains=None, include_redirect_sources=True, include_reserved_hosts=True, max_redirect_fetches=None, **kwargs)[source]
Discovers original post links.
This is a variation on http://indiewebcamp.com/original-post-discovery . It differs in that it finds multiple candidate links instead of one, and it doesn’t bother looking for MF2 (etc) markup because the silos don’t let you input it. More background: https://github.com/snarfed/bridgy/issues/51#issuecomment-136018857
Original post candidates come from the
upstreamDuplicates
,attachments
, andtags
fields, as well as links and permashortlinks/permashortcitations in the text content.- Parameters:
activity (dict) – activity
domains (sequence of str) – these domains will be considered original and stored in
upstreamDuplicates
. (Permashortcitations are exempt.)include_redirect_sources (bool) – whether to include URLs that redirect as well as their final destination URLs
include_reserved_hosts (bool) – whether to include domains on reserved TLDs (eg foo.example) and local hosts (eg http://foo.local/, http://my-server/)
max_redirect_fetches (int) – if specified, only make up to this many HTTP fetches to resolve redirects.
kwargs – passed to
requests.head()
when following redirects
- Returns:
(original post URLs, mentions)
- Return type:
- granary.as1.prefix_urls(activity, field, prefix)[source]
Adds a prefix to all matching URL fields, eg to inject a caching proxy.
Generally used with the
image
orstream
fields. For example:>>> prefix_urls({'actor': {'image': 'http://image'}}, 'image', 'https://proxy/') {'actor': {'image': 'https://proxy/http://image'}}
Skips any URL fields that already start with the prefix. URLs are not URL-encoded before adding the prefix. (This is currently used with our caching-proxy Cloudflare worker and https://cloudimage.io/ , neither of which URL-decodes.)
- granary.as1.targets(obj)[source]
Collects an AS1 activity or object’s targets.
This is all ids/URLs that are direct “targets” of the activity, eg:
the post it’s replying to
the post it’s sharing
the post it’s reacting to
the actor or other object it’s tagging
the event it’s inviting someone to
the event it’s RSVPing to
the link or object it’s bookmarking
etc…
as2
Convert between ActivityStreams 1 and 2, including ActivityPub.
- granary.as2.get_urls(obj, key='url')[source]
Returns
link['href']
orlink
, for eachlink
inobj[key]
.
- granary.as2.from_as1(obj, type=None, context='https://www.w3.org/ns/activitystreams', top_level=True)[source]
Converts an ActivityStreams 1 activity or object to ActivityStreams 2.
- granary.as2.to_as1(obj, use_type=True)[source]
Converts an ActivityStreams 2 activity or object to ActivityStreams 1.
- granary.as2.is_public(activity)[source]
Returns True if the given AS2 object or activity is public or unlisted.
https://docs.joinmastodon.org/spec/activitypub/#properties-used
- Parameters:
activity (dict) – AS2 activity or object
atom
Convert between ActivityStreams 1 and Atom.
Atom spec: https://tools.ietf.org/html/rfc4287 (RIP atomenabled.org)
- class granary.atom.Defaulter(init={})[source]
Bases:
defaultdict
Emulates Django template behavior that returns a special default value that can continue to be referenced when an attribute or item lookup fails. Helps avoid conditionals in the template itself.
https://docs.djangoproject.com/en/1.8/ref/templates/language/#variables
- granary.atom.activities_to_atom(activities, actor, title=None, request_url=None, host_url=None, xml_base=None, rels=None, reader=True)[source]
Converts ActivityStreams 1 activities to an Atom feed.
- Parameters:
actor (dict) – ActivityStreams actor, the author of the feed
title (str) – the feed <title> element. Defaults to
User feed for [NAME]
request_url (str) – URL of this Atom feed, if any. Used in a link rel=”self”.
host_url (str) – home URL for this Atom feed, if any. Used in the top-level feed
<id>
element.xml_base (str) – base URL, if any. Used in the top-level
xml:base
attribute.rels (dict) – rel links to include. Keys are string ``rel``s, values are string URLs.
reader (bool) – whether the output will be rendered in a feed reader. Currently just includes location if True, not otherwise.
- Returns:
Atom XML
- Return type:
- granary.atom.activity_to_atom(activity, xml_base=None, reader=True)[source]
Converts a single ActivityStreams 1 activity to an Atom entry.
Kwargs are passed through to
activities_to_atom()
.
- granary.atom.atom_to_activities(atom)[source]
Converts an Atom feed to ActivityStreams 1 activities.
- granary.atom.atom_to_activity(atom)[source]
Converts an Atom entry to an ActivityStreams 1 activity.
bluesky
Bluesky source class.
- granary.bluesky.url_to_did_web(url)[source]
Converts a URL to a
did:web
.In AT Proto, only hostname-based web DIDs are supported. Paths are not supported, and will be discarded.
Examples:
https://foo.com
=>did:web:foo.com
https://foo.com:3000
=>did:web:foo.com
https://foo.bar.com/baz/baj
=>did:web:foo.bar.com
- granary.bluesky.did_web_to_url(did)[source]
Converts a did:web to a URL.
In AT Proto, only hostname-based web DIDs are supported. Paths are not supported, and will throw an invalid error.
Examples:
did:web:foo.com
=>https://foo.com
did:web:foo.com%3A3000
=> INVALIDdid:web:bar.com:baz:baj
=> INVALID
- granary.bluesky.at_uri_to_web_url(uri, handle=None)[source]
Converts an at:// URI to a https://bsky.app URL.
https://atproto.com/specs/at-uri-scheme
- Parameters:
uri (str) –
at://
URIhandle – (str): optional user handle. If not provided, defaults to the DID in uri.
- Returns:
https://bsky.app URL, or None
- Return type:
- Raises:
ValueError – if uri is not a string or doesn’t start with
at://
- granary.bluesky.web_url_to_at_uri(url, handle=None)[source]
Converts a https://bsky.app URL to an
at://
URI.https://atproto.com/specs/at-uri-scheme
Currently supports profile, post, and feed URLs with DIDs and handles, eg:
- Parameters:
url (str) – bsky.app URL
- Returns:
at://
URI, or None- Return type:
- Raises:
ValueError – if url is not a string or can’t be parsed as a
bsky.app
profile or post URL
- granary.bluesky.from_as1(obj, from_url=None)[source]
Converts an AS1 object to a Bluesky object.
The
objectType
field is required.- Parameters:
- Returns:
app.bsky.*
object- Return type:
- Raises:
ValueError – if the
objectType
orverb
fields are missing or unsupported
- granary.bluesky.as1_to_profile(actor)[source]
Converts an AS1 actor to a Bluesky
app.bsky.actor.profile
.- Parameters:
actor (dict) – AS1 actor
- Raises:
ValueError – if
actor['objectType']
is not in :const:as1.ACTOR_TYPES
- granary.bluesky.to_as1(obj, type=None, repo_did=None, repo_handle=None, pds='https://bsky.social/')[source]
Converts a Bluesky object to an AS1 object.
- Parameters:
obj (dict) –
app.bsky.*
objecttype (str) – optional
$type
to parse with, only used ifobj['$type']
is unsetrepo_did (str) – optional DID of the repo this object is from. Required to generate image URLs.
repo_handle (str) – optional handle of the user whose repo this object is from
pds (str) – base URL of the PDS that currently serves this object’s repo. Required to generate image URLs. Defaults to
https://bsky.social/
.
- Returns:
AS1 object
- Return type:
- Raises:
ValueError – if the
$type
field is missing or unsupported
- granary.bluesky.blob_to_url(*, blob, repo_did, pds='https://bsky.social/')[source]
Generates a URL for a blob.
Supports both new and old style blobs: https://atproto.com/specs/data-model#blob-type
The resulting URL is a
com.atproto.sync.getBlob
XRPC call to the PDS.For blobs on the official bsky.social PDS, we could consider using their CDN instead:
https://av-cdn.bsky.app/img/avatar/plain/[DID]/[CID]@jpeg
They also run a resizing image proxy on
cdn.bsky.social
with URLs likehttps://cdn.bsky.social/imgproxy/[SIG]/rs:fit:2000:2000:1:0/plain/[CID]@jpeg
, not sure how to generate signatures for it yet.
- class granary.bluesky.Bluesky(handle, did=None, access_token=None, app_password=None)[source]
Bases:
Source
Bluesky source class. See file docstring and
Source
class for details.- __init__(handle, did=None, access_token=None, app_password=None)[source]
Constructor.
Either access_token or app_password may be provided, optionally, but not both.
- get_activities_response(user_id=None, group_id=None, app_id=None, activity_id=None, fetch_replies=False, fetch_likes=False, fetch_shares=False, include_shares=True, fetch_events=False, fetch_mentions=False, search_query=None, start_index=None, count=None, cache=None, **kwargs)[source]
Fetches posts and converts them to AS1 activities.
See
Source.get_activities_response()
for more information.Bluesky-specific details:
- Parameters:
activity_id (str) – an
at://
URI
facebook
Facebook source class. Supports both Graph API and scraping HTML.
https://developers.facebook.com/docs/graph-api/using-graph-api/
- class granary.facebook.FacebookId(user, post, comment)
Bases:
tuple
- comment
Alias for field number 2
- post
Alias for field number 1
- user
Alias for field number 0
- class granary.facebook.Facebook(access_token=None, user_id=None, scrape=False, cookie_c_user=None, cookie_xs=None)[source]
Bases:
Source
Facebook source class. See file docstring and
Source
for details.- __init__(access_token=None, user_id=None, scrape=False, cookie_c_user=None, cookie_xs=None)[source]
Constructor.
If an OAuth access token is provided, it will be passed on to Facebook. This will be necessary for some people and contact details, based on their privacy settings.
If
scrape
is True,cookie_c_user
andcookie_xs
must be provided.- Parameters:
access_token (str) – optional OAuth access token
user_id (str) – optional, current user’s id (either global or app-scoped)
scrape (bool) – whether to scrape
mbasic.facebook.com
’s HTML (True) or use the API (False)cookie_c_user (str) – optional
c_user
cookie to use when scrapingcookie_xs (str) – optional
xs
cookie to use when scraping
- user_url(id)
Returns the URL for a user’s profile.
- get_actor(user_id=None)[source]
Returns a user as a JSON ActivityStreams actor dict.
- Parameters:
user_id (str) – id or username. Defaults to
me
, ie the current user.
- get_activities_response(user_id=None, group_id=None, app_id=None, activity_id=None, start_index=0, count=0, etag=None, min_id=None, cache=None, fetch_replies=False, fetch_likes=False, fetch_shares=False, fetch_events=False, fetch_mentions=False, search_query=None, fetch_news=False, event_owner_id=None, **kwargs)[source]
Fetches posts and converts them to ActivityStreams activities.
See
Source.get_activities_response()
for details.Likes, top-level replies (ie comments), and reactions are always included. They come from the
comments
,likes
, andreactions
fields in the Graph API’s Post object: https://developers.facebook.com/docs/reference/api/post/Threaded comments, ie comments in reply to other top-level comments, require an additional API call, so they’re only included if fetch_replies is True.
Mentions are never fetched or included because the API doesn’t support searching for them. https://github.com/snarfed/bridgy/issues/523#issuecomment-155523875
Most parameters are documented in
Source.get_activities_response()
. Additional parameters are documented here.- Parameters:
fetch_news (bool) – whether to also fetch and include Open Graph news stories (
/USER/news.publishes
). Requires theuser_actions.news
permission. Background in https://github.com/snarfed/bridgy/issues/479event_owner_id (str) – if provided, only events owned by this user id will be returned. Avoids (but doesn’t entirely prevent) processing big non-indieweb events with tons of attendees that put us over App Engine’s instance memory limit. https://github.com/snarfed/bridgy/issues/77
- get_comment(comment_id, activity_id=None, activity_author_id=None, activity=None)[source]
Returns an ActivityStreams comment object.
Returns an ActivityStreams share activity object.
- get_reaction(activity_user_id, activity_id, reaction_user_id, reaction_id, activity=None)[source]
Fetches and returns a reaction.
- Parameters:
activity_user_id (str) – id of the user who posted the original activity
activity_id (str) – activity id
reaction_user_id (str) – id of the user who reacted
reaction_id (str) – id of the reaction. one of:
love
,wow
,haha
,sad
,angry
,thankful
,pride
,care
activity (dict) – activity object (optional)
- Return type:
- create(obj, include_link='omit', ignore_formatting=False)[source]
Creates a new post, comment, like, or RSVP.
- Parameters:
- Returns:
contents will be a dict with
id
andurl
keys for the newly created Facebook object- Return type:
CreationResult or None
- preview_create(obj, include_link='omit', ignore_formatting=False)[source]
Previews creating a new post, comment, like, or RSVP.
- Parameters:
- Returns:
contents will be an HTML snippet
- Return type:
CreationResult or None
- create_notification(user_id, text, link)[source]
Sends the authenticated user a notification.
Uses the Notifications API (beta): https://developers.facebook.com/docs/games/notifications/#impl
- Parameters:
user_id (str) – username or user ID
text (str) – shown to the user in the notification
link (str) – relative URL, the user is redirected here when they click on the notification. Note that only the path and query parameters are used! they’re combined with the domain in your Facebook app’s Game App URL: https://developers.facebook.com/docs/games/services/appnotifications#parameters
- Raises:
urllib3.HTPPError –
- post_url(post)[source]
Returns a short Facebook URL for a post.
- Parameters:
post (dict) – Facebook post
- comment_url(post_id, comment_id, post_author_id=None)[source]
Returns a short Facebook URL for a comment.
- classmethod base_id(url)[source]
Guesses the id of the object in the given URL.
- Return type:
str or None
- base_object(obj, verb=None, resolve_numeric_id=False)[source]
Returns the ‘base’ silo object that an object operates on.
This is mostly a big bag of heuristics for reverse engineering and parsing Facebook URLs. Whee.
- Parameters:
- Returns:
minimal ActivityStreams object. Usually has at least
id
,numeric_id
, andurl
fields; may also haveauthor
.- Return type:
- post_to_object(post, type=None)[source]
Converts a post to an object.
TODO: handle the
sharedposts
field
- comment_to_object(comment, post_id=None, post_author_id=None)[source]
Converts a comment to an object.
- Parameters:
- Returns:
ActivityStreams object
- Return type:
Converts a share (from
/OBJECT/sharedposts
) to an object.
- rsvp_to_object(rsvp, type=None, event=None)[source]
Converts an RSVP to an object.
The
id
field will ony be filled in ifevent['id']
is provided.
- privacy_to_to(obj, type=None)[source]
Converts a Facebook
privacy
field to an ActivityStreamsto
field.privacy
is sometimes an object: https://developers.facebook.com/docs/graph-api/reference/post#fields…and other times a string: https://developers.facebook.com/docs/graph-api/reference/album/#readfields
- email_to_object(html)[source]
Converts a Facebook HTML notification email to an AS1 object.
- Parameters:
html – str
- Returns:
ActivityStreams object, or None if
html
couldn’t be parsed- Return type:
- scraped_to_activities(scraped, log_html=False, **kwargs)[source]
Converts HTML from an
mbasic.facebook.com
timeline to AS1 activities.
- scraped_to_activity(scraped, log_html=False, **kwargs)[source]
Converts HTML from an
mbasic.facebook.com
post page to an AS1 activity.
- merge_scraped_reactions(scraped, activity)[source]
Converts and merges scraped likes and reactions into an activity.
New likes and emoji reactions are added to the activity in
tags
. Existing likes and emoji reactions intags
are ignored.
- static parse_id(id, is_comment=False)[source]
Parses a Facebook post or comment id.
Facebook ids come in different formats:
Simple number, usually a user or post:
12
Two numbers with underscore, usually
POST_COMMENT
orUSER_POST
:12_34
Three numbers with underscores,
USER_POST_COMMENT
:12_34_56
Three numbers with colons,
USER:POST:SHARD
:12:34:63
(We’re guessing that the third part is a shard in some FB internal system. In our experience so far, it’s always either 63 or the app-scoped user id for 63.)Two numbers with colon,
POST:SHARD
:12:34
(We’ve seen 0 as shard in this format.)Four numbers with colons/underscore,
USER:POST:SHARD_COMMENT
:12:34:63_56
Five numbers with colons/underscore,
USER:EVENT:UNKNOWN:UNKNOWN_UNKNOWN
Not currently supported! Examples:111599105530674:998145346924699:10102446236688861:10207188792305341_998153510257216
111599105530674:195181727490727:10102446236688861:10205257726909910_195198790822354
Background:
- Parameters:
- Returns:
Some or all fields may be None.
- Return type:
- resolve_object_id(user_id, post_id, activity=None)[source]
Resolve a post id to its Facebook object id, if any.
Used for photo posts, since Facebook has (at least) two different objects (and ids) for them, one for the post and one for each photo.
This is the same logic that we do for canonicalizing photo objects in
get_activities()
above.If activity is not provided, fetches the post from Facebook.
- urlopen(url, _as=<class 'dict'>, **kwargs)[source]
Wraps
urllib2.urlopen()
and passes through the access token.
- urlopen_batch(urls)[source]
Sends a batch of multiple API calls using Facebook’s batch API.
Raises the appropriate
urllib2.HTTPError
if any individual call returns HTTP status code 4xx or 5xx.https://developers.facebook.com/docs/graph-api/making-multiple-requests
- urlopen_batch_full(requests)[source]
Sends a batch of multiple API calls using Facebook’s batch API.
Similar to
urlopen_batch()
, but the requests arg and return value are dicts with headers, HTTP status code, etc. Raisesurllib2.HTTPError
if the outer batch request itself returns an HTTP error.https://developers.facebook.com/docs/graph-api/making-multiple-requests
- Parameters:
requests (sequence of dict) –
requests in Facebook’s batch format, except that headers is a single dict, not a list of dicts, e.g.:
[{'relative_url': 'me/feed', 'headers': {'ETag': 'xyz', ...}, }, ... ]
- Returns:
responses in Facebook’s batch format, except that body is JSON-decoded if possible, and headers is a single dict, not a list of dicts, e.g.:
[{'code': 200, 'headers': {'ETag': 'xyz', ...}, 'body': {...}, }, ... ]
- Return type:
sequence of dict
flickr
Flickr source class.
Uses Flickr’s REST API https://www.flickr.com/services/api/
TODO: Fetching feeds with comments and/or favorites is very request
intensive right now. It would be ideal to find a way to batch
requests, make requests asynchronously, or make better calls to the
API itself. Maybe use flickr.activity.userPhotos
(https://www.flickr.com/services/api/flickr.activity.userPhotos.html)
when group_id=SELF`.
- class granary.flickr.Flickr(access_token_key, access_token_secret, user_id=None, path_alias=None)[source]
Bases:
Source
Flickr source class. See file docstring and Source class for details.
- __init__(access_token_key, access_token_secret, user_id=None, path_alias=None)[source]
Constructor.
If they are not provided,
user_id
andpath_alias
will be looked up via the API on first use.
- create(obj, include_link='omit', ignore_formatting=False)[source]
Creates a photo, comment, or favorite.
- Parameters:
- Returns:
content
will be a dict withid
,url
, andtype
keys (all optional) for the newly created Flickr object- Return type:
CreationResult or None
- preview_create(obj, include_link='omit', ignore_formatting=False)[source]
Preview creation of a photo, comment, or favorite.
- Parameters:
- Returns:
whose description will be an HTML summary of what publishing will do, and whose content will be an HTML preview of the result
- Return type:
CreationResult or None
- delete(id)[source]
Deletes a photo. The authenticated user must have created it.
- Parameters:
- Returns:
content
is Flickr API response dict- Return type:
- get_activities_response(user_id=None, group_id=None, app_id=None, activity_id=None, start_index=0, count=0, etag=None, min_id=None, cache=None, fetch_replies=False, fetch_likes=False, fetch_shares=False, fetch_events=False, fetch_mentions=False, search_query=None, **kwargs)[source]
Fetches Flickr photos and converts them to ActivityStreams activities.
See
Source.get_activities_response()
for details.Mentions are not fetched or included because they don’t exist in Flickr. https://github.com/snarfed/bridgy/issues/523#issuecomment-155523875
- get_actor(user_id=None)[source]
Get an ActivityStreams object of type
person
given a user’s nsid. If nouser_id
is provided, this method will make another API request to find out the currently logged in user’s id.
- get_comment(comment_id, activity_id=None, activity_author_id=None, activity=None)[source]
Returns an ActivityStreams comment object.
- photo_to_activity(photo)[source]
Convert a Flickr photo to an ActivityStreams object.
Takes either data in the expanded form returned by
flickr.photos.getInfo
or the abbreviated form returned byflickr.people.getPhotos
.
- like_to_object(person, photo_activity)[source]
Convert a Flickr favorite into an ActivityStreams
like
tag.
- comment_to_object(comment, photo_id)[source]
Convert a Flickr comment JSON object to an ActivityStreams comment.
- get_user_image(farm, server, author)[source]
Convert fields from a typical Flickr response into the buddy icon URL.
- user_id()[source]
Get the nsid of the currently authorized user.
The first time this is called, it will invoke the
flickr.people.getLimits
API method.https://www.flickr.com/services/api/flickr.people.getLimits.html
- Return type:
- path_alias()[source]
Get the path_alias of the currently authorized user.
The first time this is called, it will invoke the
flickr.people.getInfo
API method.https://www.flickr.com/services/api/flickr.people.getInfo.html
- Return type:
github
GitHub source class. Uses the v4 GraphQL API and the v3 REST API.
API docs:
- class granary.github.GitHub(access_token=None)[source]
Bases:
Source
GitHub source class. See file docstring and
Source
for details.- __init__(access_token=None)[source]
Constructor.
- Parameters:
access_token (str) – optional OAuth access token
- rest(url, data=None, parse_json=True, **kwargs)[source]
Makes a v3 REST API call.
Uses HTTP POST if data is provided, otherwise GET.
- Parameters:
data (dict) – JSON payload for POST requests
json (bool) – whether to parse the response body as JSON and return it as a dict. If False, returns a
requests.Response
instead.
- Returns:
- decoded from JSON response if
json=True
, otherwise
- decoded from JSON response if
- Return type:
- get_activities_response(user_id=None, group_id=None, app_id=None, activity_id=None, start_index=0, count=0, etag=None, min_id=None, cache=None, fetch_replies=False, fetch_likes=False, fetch_shares=False, fetch_events=False, fetch_mentions=False, search_query=None, public_only=True, **kwargs)[source]
Fetches issues and comments and converts them to ActivityStreams activities.
See
Source.get_activities_response()
for details.Not comprehensive! Uses the notifications API (v3 REST).
Also note that start_index is not currently supported.
fetch_likes
determines whether emoji reactions are fetched: https://help.github.com/articles/about-conversations-on-github#reacting-to-ideas-in-commentsThe notifications API call supports
Last-Modified
/If-Modified-Since
headers and304 Not Changed
responses. If provided,etag
should be an RFC2822 timestamp, usually the exact value returned in aLast-Modified
header. It will also be passed to the comments API endpoint as thesince=
value (converted to ISO8601).
- render_markdown(markdown, owner, repo)[source]
Uses the GitHub API to render GitHub-flavored Markdown to HTML.
- create(obj, include_link='omit', ignore_formatting=False)[source]
Creates a new issue or comment.
- Parameters:
- Returns:
contents will be a dict with
id
andurl
keys for the newly created GitHub object- Return type:
CreationResult or None
- preview_create(obj, include_link='omit', ignore_formatting=False)[source]
Previews creating an issue or comment.
- Parameters:
- Returns:
contents
will be a str HTML snippet- Return type:
CreationResult or None
- issue_to_object(issue)[source]
Converts a GitHub issue or pull request to ActivityStreams.
Handles both v4 GraphQL and v3 REST API issue and PR objects.
- pr_to_object(issue)
Converts a GitHub issue or pull request to ActivityStreams.
Handles both v4 GraphQL and v3 REST API issue and PR objects.
- comment_to_object(comment)[source]
Converts a GitHub comment to ActivityStreams.
Handles both v4 GraphQL and v3 REST API issue objects.
- reaction_to_object(reaction, target)[source]
Converts a GitHub emoji reaction to ActivityStreams.
Handles v3 REST API reaction objects.
instagram
Instagram source class.
Instagram’s API doesn’t tell you if a user has marked their account private or
not, so the Audience Targeting to
field is currently always set to
@public
.
https://groups.google.com/forum/m/#!topic/instagram-api-developers/DAO7OriVFsw
https://groups.google.com/forum/#!searchin/instagram-api-developers/private
- class granary.instagram.Instagram(access_token=None, allow_comment_creation=False, scrape=False, cookie=None)[source]
Bases:
Source
Instagram source class. See file docstring and Source class for details.
- __init__(access_token=None, allow_comment_creation=False, scrape=False, cookie=None)[source]
Constructor.
If an OAuth access token is provided, it will be passed on to Instagram. This will be necessary for some people and contact details, based on their privacy settings.
- Parameters:
access_token (str) – optional OAuth access token
allow_comment_creation (bool) – optionally disable comment creation, useful if the app is not approved to create comments.
scrape (bool) – whether to scrape instagram.com’s HTML (True) or use the API (False)
cookie (str) – optional sessionid cookie to use when scraping.
- get_actor(user_id=None, **kwargs)[source]
Returns a user as a JSON ActivityStreams actor dict.
- Parameters:
user_id (str) – id or username. Defaults to
self
, ie the current user.kwargs – if scraping, passed through to :meth:get_activities_response``.
- Raises:
AssertionError – if kwargs is provided but we’re not scraping
- get_activities_response(user_id=None, group_id=None, app_id=None, activity_id=None, start_index=0, count=0, etag=None, min_id=None, cache=None, fetch_replies=False, fetch_likes=False, fetch_shares=False, fetch_events=False, fetch_mentions=False, search_query=None, scrape=False, cookie=None, ignore_rate_limit=False, **kwargs)[source]
Fetches posts and converts them to ActivityStreams activities.
See
Source.get_activities_response()
for details.app_id
is ignored. Supportsmin_id
, but notETag
, since Instagram doesn’t support it.http://instagram.com/developer/endpoints/users/#get_users_feed
http://instagram.com/developer/endpoints/users/#get_users_media_recent
Likes are always included, regardless of the
fetch_likes
kwarg. They come bundled in thelikes
field of the API Media object: http://instagram.com/developer/endpoints/media/#Mentions are never fetched or included because the API doesn’t support searching for them. https://github.com/snarfed/bridgy/issues/523#issuecomment-155523875
Shares are never fetched or included since there is no share feature.
Instagram only supports search over hashtags, so if search_query is set, it must begin with
#
.May populate a custom
ig_like_count
property in media objects. (Currently only when scraping.)- Parameters:
scrape (bool) – if True, scrapes HTML from instagram.com instead of using the API. Populates the user’s actor object in the
actor
response field. Useful for apps that haven’t yet been approved in the new permissions approval process. Currently only supportsgroup_id=SELF
. Also supports passing a shortcode as activity_id as well as the internal API id. http://developers.instagram.com/post/133424514006/instagram-platform-updatecookie (str) – only used if
scrape=True
ignore_rate_limit (bool) – for scraping, always make an HTTP request, even if we’ve been rate limited recently
- get_comment(comment_id, activity_id=None, activity_author_id=None, activity=None)[source]
Returns an ActivityStreams comment object.
Not implemented. Returns None. Resharing isn’t a feature of Instagram.
- create(obj, include_link='omit', ignore_formatting=False)[source]
Creates a new comment or like.
- Parameters:
- Returns:
- if successful, content will have and
id
andurl
keys for the newly created Instagram object
- if successful, content will have and
- Return type:
- preview_create(obj, include_link='omit', ignore_formatting=False)[source]
Preview a new comment or like.
- Parameters:
- Returns:
- if successful, content and description will describe the
new Instagram object.
- Return type:
- base_object(obj)[source]
Extends the default base_object() to avoid using shortcodes as object ids.
- static id_to_shortcode(id)[source]
Converts a media id to the shortcode used in its instagram.com URL.
Based on http://carrot.is/coding/instagram-ids , which determined that shortcodes are just URL-safe base64 encoded ids.
- scraped_to_activities(input, cookie=None, count=None, fetch_extras=False)[source]
Converts scraped Instagram HTML to ActivityStreams activities.
The input HTML may be from:
a user’s feed, eg https://www.instagram.com/ while logged in
a user’s profile, eg https://www.instagram.com/snarfed/
a photo or video, eg https://www.instagram.com/p/BBWCSrfFZAk/
serialized JSON from the API for a feed, profile, or post, eg https://i.instagram.com/api/v1/feed/timeline/
- Parameters:
- Returns:
([ActivityStreams activities], ActivityStreams viewer actor)
- Return type:
- scraped_json_to_activities(input, cookie=None, count=None, fetch_extras=False)[source]
Converts scraped Instagram JSON to ActivityStreams activities.
- The input JSON may be from a user’s profile, eg
https://i.instagram.com/api/v1/users/web_profile_info/?username=…
- Parameters:
- Returns:
([ActivityStreams activities], ActivityStreams viewer actor)
- Return type:
- html_to_activities(input, cookie=None, count=None, fetch_extras=False)
Converts scraped Instagram HTML to ActivityStreams activities.
The input HTML may be from:
a user’s feed, eg https://www.instagram.com/ while logged in
a user’s profile, eg https://www.instagram.com/snarfed/
a photo or video, eg https://www.instagram.com/p/BBWCSrfFZAk/
serialized JSON from the API for a feed, profile, or post, eg https://i.instagram.com/api/v1/feed/timeline/
- Parameters:
- Returns:
([ActivityStreams activities], ActivityStreams viewer actor)
- Return type:
- scraped_to_activity(html, **kwargs)[source]
Converts HTML from photo/video permalink page to an AS1 activity.
- scraped_to_actor(html, **kwargs)[source]
Extracts and returns the logged in actor from any Instagram HTML.
- merge_scraped_comments(scraped, activity)[source]
Converts and merges scraped comments (replies) into an activity.
- Parameters:
- Returns:
AS comment objects converted from scraped
- Return type:
- Raises:
ValueError – if scraped is not valid JSON
jsonfeed
Convert between ActivityStreams and JSON Feed.
JSON Feed spec: https://jsonfeed.org/version/1
- granary.jsonfeed.activities_to_jsonfeed(activities, actor=None, title=None, feed_url=None, home_page_url=None)[source]
Converts ActivityStreams activities to a JSON feed.
- Parameters:
- Returns:
JSON Feed data
- Return type:
- granary.jsonfeed.jsonfeed_to_activities(jsonfeed)[source]
Converts a JSON feed to ActivityStreams activities and actor.
- Parameters:
jsonfeed (dict) – JSON Feed data
- Returns:
(activities, actor)
, where activities and actor are both ActivityStreams object dicts- Return type:
- Raises:
ValueError – if jsonfeed isn’t a valid JSON Feed dict
mastodon
Mastodon source class.
Mastodon is an ActivityPub implementation, but it also has a REST + OAuth 2 API independent of AP. This class handles that API. API docs: https://docs.joinmastodon.org/api/
May also be used for services with Mastodon-compatible APIs, eg Pleroma: https://docs-develop.pleroma.social/backend/API/differences_in_mastoapi_responses/
- class granary.mastodon.Mastodon(instance, access_token, user_id=None, truncate_text_length=None)[source]
Bases:
Source
Mastodon source class. See file docstring and Source class for details.
- __init__(instance, access_token, user_id=None, truncate_text_length=None)[source]
Constructor.
If
user_id
is not provided, it will be fetched via the API.- Parameters:
instance (str) – base URL of Mastodon instance, eg https://mastodon.social/
user_id – (str or int): optional, current user’s id (not username!) on this instance
access_token (str) – optional OAuth access token
truncate_text_length (int) – optional character limit for toots, overrides the default of 500
- get_activities_response(user_id=None, group_id=None, app_id=None, activity_id=None, fetch_replies=False, fetch_likes=False, fetch_shares=False, include_shares=True, fetch_events=False, fetch_mentions=False, search_query=None, start_index=0, count=0, cache=None, **kwargs)[source]
Fetches toots and converts them to ActivityStreams activities.
See
Source.get_activities_response()
for details.
- get_comment(comment_id, **kwargs)[source]
Fetches and returns a comment.
- Parameters:
comment_id (str) – status id
kwargs – unused
- Returns:
ActivityStreams object
- Return type:
- Raises:
ValueError – if comment_id is invalid
- create(obj, include_link='omit', ignore_formatting=False)[source]
Creates a status (aka toot), reply, boost (aka reblog), or favorite.
https://docs.joinmastodon.org/methods/statuses/
- Parameters:
- Returns:
whose content will be a dict with
id
,url
, andtype
keys (all optional) for the newly created object (or None)- Return type:
- preview_create(obj, include_link='omit', ignore_formatting=False)[source]
Preview creating a status (aka toot), reply, boost (aka reblog), or favorite.
https://docs.joinmastodon.org/methods/statuses/
- Parameters:
- Returns:
content will be a str HTML snippet or None
- Return type:
- base_object(obj)[source]
Returns the “base” Mastodon object that an object operates on.
If the object is a reply, boost, or favorite of a Mastodon post - on any instance - this returns that post object. The id in the returned object is the id of that remote post on the local instance. (As a Mastodon style id, ie an int in a string, not a tag URI.)
Uses Mastodon’s search API on the local instance to determine whether a URL is a Mastodon post, and if it is, to find or generate an id for it on the local instance.
Discovered via https://mastodon.social/@jkreeftmeijer/101245063526942536
- delete(id)[source]
Deletes a toot. The authenticated user must have authored it.
- Parameters:
- Returns:
content is dict with
url
andid
fields- Return type:
- get_blocklist_ids()[source]
Returns the current user’s block list as a list of int account ids.
May make multiple API calls to fully fetch large block lists. https://docs.joinmastodon.org/methods/accounts/blocks/
- Returns:
Mastodon account ids on the current instance
- Return type:
sequence of int
meetup
Meetup.com source class.
- class granary.meetup.Meetup(access_token)[source]
Bases:
Source
- create(obj, include_link='omit', ignore_formatting=False)[source]
Creates a new object: a post, comment, like, share, or RSVP.
Subclasses should override this. Different sites will support different functionality; check each subclass for details. The actor will usually be the authenticated user.
- Parameters:
obj (dict) – ActivityStreams object. At minimum, must have the content field. objectType is strongly recommended.
include_link (str) –
INCLUDE_LINK
,OMIT_LINK
, orINCLUDE_IF_TRUNCATED
; whether to include a link to the object (if it has one) in the content.ignore_formatting (bool) – whether to use content text as is, instead of converting its HTML to plain text styling (newlines, etc.)
- Returns:
The result.
content
will be a dict or None. If the newly created object has an id or permalink, they’ll be provided in the values forid
andurl
.- Return type:
- preview_create(obj, include_link='omit', ignore_formatting=False)[source]
Previews creating a new object: a post, comment, like, share, or RSVP.
Returns HTML that previews what
create()
with the same object will do.Subclasses should override this. Different sites will support different functionality; check each subclass for details. The actor will usually be the authenticated user.
- Parameters:
obj (dict) – ActivityStreams object. At minimum, must have the
content
field.objectType
is strongly recommended.include_link (str) –
INCLUDE_LINK
,OMIT_LINK
, orINCLUDE_IF_TRUNCATED
; whether to include a link to the object (if it has one) in the content.ignore_formatting (bool) – whether to use content text as is, instead of converting its HTML to plain text styling (newlines, etc.)
- Returns:
The result. content will be a dict or
None
.- Return type:
microformats2
Convert ActivityStreams to microformats2 HTML and JSON.
Microformats2 specs: http://microformats.org/wiki/microformats2
ActivityStreams 1 specs: http://activitystrea.ms/specs/
- granary.microformats2.get_string_urls(objs)[source]
Extracts string URLs from a list of either string URLs or mf2 dicts.
Many mf2 properties can contain either string URLs or full mf2 objects, eg
h-cite
.in-reply-to
is the most commonly used example: http://indiewebcamp.com/in-reply-to#How_to_consume_in-reply-to
- granary.microformats2.get_text(val)[source]
Returns a plain text string value. See
get_html()
.
- granary.microformats2.maybe_normalize_iso8601(val)[source]
Tries to normalize a string datetime value to ISO-8601.
- granary.microformats2.activity_to_json(activity, **kwargs)[source]
Converts an ActivityStreams activity to microformats2 JSON.
- Parameters:
activity (dict) – a decoded JSON ActivityStreams activity
kwargs – passed to :fun:`object_to_json`
- Returns:
decoded microformats2 JSON
- Return type:
- granary.microformats2.object_to_json(obj, trim_nulls=True, entry_class='h-entry', default_object_type=None, synthesize_content=True)[source]
Converts an ActivityStreams object to microformats2 JSON.
- Parameters:
obj (dict) – a decoded JSON ActivityStreams object
trim_nulls (bool) – whether to remove elements with null or empty values
entry_class (str or sequence of str), the mf2 class(es) – given, eg
h-cite
when parsing a reference to a foreign entry. defaults to h-entry`default_object_type (str) – the ActivityStreams
objectType
to use if one is not present. defaults to Nonesynthesize_content (bool) – whether to generate synthetic content if the object doesn’t have its own, eg
likes this
orshared this
- Returns:
decoded microformats2 JSON
- Return type:
- granary.microformats2.json_to_object(mf2, actor=None, fetch_mf2=False, rel_urls=None)[source]
Converts a single microformats2 JSON item to an ActivityStreams object.
Supports
h-entry
,h-event
,h-card
, and other single item times. Does not yet supporth-feed
.If
rel_urls
is provided, the returnedurl
andurls
fields will be objects that may includedisplayName
fields with the text ortitle
from the original HTML links.- Parameters:
mf2 (dict) – decoded JSON microformats2 object
actor (dict) – optional author AS actor object. usually comes from a
rel="author"
link. ifmf2
has its own author, that overrides thisfetch_mf2 (bool) – whether to fetch additional pages via HTTP if necessary, eg to determine authorship: https://indieweb.org/authorship
rel_urls (dict) – optional rel-urls field from parsed mf2
- Returns:
ActivityStreams object
- Return type:
- granary.microformats2.html_to_activities(html, url=None, actor=None, id=None)[source]
Converts a microformats2 HTML
h-feed
to ActivityStreams activities.- Parameters:
html (str) – HTML or
requests.Response
url (str) – optional URL that HTML came from
actor (dict) – optional author AS actor object for all activities. usually comes from a
rel="author"
link.id (str) – optional id of specific element to extract and parse. defaults to the whole page.
- Returns:
ActivityStreams activities
- Return type:
- granary.microformats2.json_to_activities(parsed, actor=None)[source]
Converts parsed microformats2 JSON to ActivityStreams activities.
- granary.microformats2.activities_to_html(activities, extra='', body_class='')[source]
Converts ActivityStreams activities to a microformats2 HTML
h-feed
.- Parameters:
- Returns:
the content field in
obj
with the tags in thetags
field converted to links if they havestartIndex
andlength
, otherwise added to the end.- Return type:
- granary.microformats2.object_to_html(obj, parent_props=None, synthesize_content=True)[source]
Converts an ActivityStreams object to microformats2 HTML.
Features:
linkifies embedded tags and adds links for other tags
linkifies embedded URLs
adds links, summaries, and thumbnails for attachments and checkins
adds a “via SOURCE” postscript
- Parameters:
- Returns:
the content field in
obj
with tags in thetags
field converted to links if they havestartIndex
andlength
, otherwise added to the end.- Return type:
- granary.microformats2.json_to_html(obj, parent_props=None)[source]
Converts a microformats2 JSON object to microformats2 HTML.
See
object_to_html()
for details.
- granary.microformats2.render_content(obj, include_location=True, synthesize_content=True, render_attachments=False, render_image=False, white_space_pre=True)[source]
Renders the content of an ActivityStreams object as HTML.
Includes tags, mentions, and non-note/article attachments. (Note/article attachments are converted to mf2 children in object_to_json and then rendered in
json_to_html()
.)- Parameters:
obj (dict) – decoded JSON ActivityStreams object
include_location (bool) – whether to render location, if provided
synthesize_content (bool) – whether to generate synthetic content if the object doesn’t have its own, eg
likes this
orshared this
render_attachments (bool) – whether to render attachments, eg links, images, audio, and video
render_image (bool) – whether to render the object’s image(s)
white_space_pre (bool) – whether to wrap in CSS
white-space: pre
. If False, newlines will be converted to<br>
tags instead. Background: https://indiewebcamp.com/note#Indieweb_whitespace_thinking
- Returns:
rendered HTML
- Return type:
- granary.microformats2.find_author(parsed, **kwargs)[source]
Returns the author of a page as a ActivityStreams actor.
- granary.microformats2.tags_to_html(tags, classname, visible=True)[source]
Returns an HTML string with links to the given tag objects.
- granary.microformats2.author_display_name(hcard)[source]
Returns a human-readable string display name for an
h-card
object.
- granary.microformats2.maybe_linked_name(props)[source]
Returns the HTML for a
p-name
with an optionalu-url
inside.
- granary.microformats2.img(src, alt='')[source]
Returns an
<img>
str with the givensrc
,class
, andalt
.
- granary.microformats2.vid(src, poster='')[source]
Returns an
<video>
str with the givensrc
andposter
.
- granary.microformats2.maybe_linked(text, url=None, linked_classname=None, unlinked_classname=None)[source]
Wraps text in an
<a href=...>
iff a non-empty url is provided.
nostr
Nostr.
NIPS implemented:
01: base protocol, events, profile metadata
02: contacts/followings
05: domain identifiers
09: deletes
10: replies, mentions
12: hashtags, locations
14: subject tag in notes
18: reposts, including 10 for e/p tags
19: bech32-encoded ids
21: nostr: URI scheme
23: articles
25: likes, emoji reactions
27: text notes
39: external identities
50: search
TODO:
05: DNS verification
11: relay info (like nodeinfo)
12: tag queries
16, 33: ephemeral/replaceable events
27: user mentions, note/event mentions
the difficulty is that the Nostr tags don’t include human-readable
text. clients are supposed to get that from their local database.
32: tag activities
46: “Nostr Connect,” signing proxy that holds user’s keys
65: user relays. what would this be in AS1? anything?
- granary.nostr.uri_to_id(uri)[source]
Converts a nostr: URI with bech32-encoded id to a hex sha256 hash id.
Based on NIP-19 and NIP-21.
- granary.nostr.id_to_uri(prefix, id)[source]
Converts a hex sha256 hash id to a nostr: URI with bech32-encoded id.
Based on NIP-19 and NIP-21.
- granary.nostr.from_as1(obj)[source]
Converts an ActivityStreams 1 activity or object to a Nostr event.
- granary.nostr.to_as1(event)[source]
Converts a Nostr event to an ActivityStreams 2 activity or object.
- class granary.nostr.Nostr(relays)[source]
Bases:
Source
Nostr source class. See file docstring and
Source
for details.- get_activities_response(user_id=None, group_id=None, app_id=None, activity_id=None, fetch_replies=False, fetch_likes=False, fetch_shares=False, include_shares=True, fetch_events=False, fetch_mentions=False, search_query=None, start_index=None, count=None, cache=None, **kwargs)[source]
Fetches events and converts them to AS1 activities.
See
Source.get_activities_response()
for more information.
- query(websocket, filter)[source]
Runs a Nostr
REQ
query on an open websocket.Sends the query, collects the responses, and closes the
REQ
subscription. Iflimit
is not set on the filter, it defaults to 20.- Parameters:
websocket (websockets.sync.client.ClientConnection) –
filter (dict) – NIP-01
REQ
filterlimit (int) –
- Returns:
Nostr events
- Return type:
- Raises:
AssertionError – if the filter
limit
field is not set.
pixelfed
Pixelfed source class, heavily based on Mastodon.
Pixelfed’s API is a clone of Mastodon’s. https://docs.pixelfed.org/technical-documentation/api-v1.html
reddit
Reddit source class.
Not thread safe!
Reddit API docs:
PRAW API docs: https://praw.readthedocs.io/
- class granary.reddit.Reddit(refresh_token)[source]
Bases:
Source
Reddit source class. See file docstring and
source.Source
for details.- praw_to_actor(praw_user)[source]
Converts a PRAW Redditor to an actor.
Makes external calls to fetch data from the Reddit API.
https://praw.readthedocs.io/en/latest/code_overview/models/redditor.html
Caches fetched user data for 5m to avoid repeating user profile API requests when fetching multiple comments or posts from the same author. Background: https://github.com/snarfed/bridgy/issues/1021
Ideally this would be part of PRAW, but they seem uninterested:
- Parameters:
user (praw.models.Redditor) –
- Returns:
ActivityStreams actor
- Return type:
- praw_to_object(thing, type)[source]
Converts a PRAW object to an AS1 object.
Currently only returns public content.
Note that this will make external API calls to lazily load some attributes.
- Parameters:
thing (praw.models.Submission or praw.models.Comment) –
type (str) – either
submission
orcomment
, which content to get
- Returns:
ActivityStreams object
- Return type:
- praw_to_activity(thing, type)[source]
Converts a PRAW submission or comment to an activity.
Note that this will make external API calls to lazily load some attributes.
https://praw.readthedocs.io/en/latest/code_overview/models/submission.html
https://praw.readthedocs.io/en/latest/code_overview/models/comment.html
- Parameters:
thing (praw.models.Submission or praw.models.Comment) –
type (str) – whether to get submission or comment content
- Returns:
ActivityStreams activity
- Return type:
- get_activities_response(user_id=None, group_id=None, app_id=None, activity_id=None, start_index=0, count=None, etag=None, min_id=None, cache=None, fetch_replies=False, fetch_likes=False, fetch_shares=False, fetch_events=False, fetch_mentions=False, search_query=None, **kwargs)[source]
Fetches submissions and ActivityStreams activities.
Currently only implements
activity_id
,search_query
andfetch_replies
.
- get_actor(user_id=None)[source]
Fetches a Reddit user and converts them to an AS1 actor.
- Parameters:
user_id (str) –
- Returns
dict: AS1 actor, or
{}
if the user isn’t found
rss
Convert between ActivityStreams and RSS 2.0.
RSS 2.0 spec: http://www.rssboard.org/rss-specification
Feedgen docs: https://feedgen.kiesow.be/
Apple iTunes Podcasts feed requirements: https://help.apple.com/itc/podcasts_connect/#/itc1723472cb
Notably:
Valid RSS 2.0.
Each podcast item requires
<guid>
.Images should be JPEG or PNG, 1400x1400 to 3000x3000.
HTTP server that hosts assets and files should support range requests.
source
Source base class.
Based on the OpenSocial ActivityStreams REST API: http://opensocial-resources.googlecode.com/svn/spec/2.0.1/Social-API-Server.xml#ActivityStreams-Service
Uses the to
field of the Audience Targeting extension to indicate an
activity’s privacy settings. It’s set to a group with alias @public
or
@private
, or unset if unknown.
http://activitystrea.ms/specs/json/targeting/1.0/#anchor3
- class granary.source.CreationResult(content, description, abort, error_plain, error_html)
Bases:
tuple
Result of creating a new object in a silo.
create()
andpreview_create()
use this to provide a detailed description of publishing failures. Ifabort
is False, we should continue looking for an entry to publish; if True, we should immediately inform the user.error_plain
text is sent in response to failed publish webmentions;error_html
will be displayed to the user when publishing interactively.- description
HTML snippet describing the publish action, e.g.
@-reply
orRSVP yes to this event
. The verb itself is surrounded by a<span class="verb">
to allow styling. May also include<a>
link(s) and embedded silo post(s).- Type:
- abort
Alias for field number 2
- content
Alias for field number 0
- description
Alias for field number 1
- error_html
Alias for field number 4
- error_plain
Alias for field number 3
- exception granary.source.RateLimited(*args, **kwargs)[source]
Bases:
BaseException
Raised when an API rate limits us, and we may have a partial result.
- partial
the partial result, if any. Usually a list.
- granary.source.html_to_text(html, baseurl='', **kwargs)[source]
Converts string html to string text with html2text.
- Parameters:
baseurl (str) – base URL to use when resolving relative URLs. Passed through to
HTML2Text
.kwargs – html2text options: https://github.com/Alir3z4/html2text/blob/master/docs/usage.md#available-options
- granary.source.load_json(body, url)[source]
Utility method to parse a JSON string. Raises HTTPError 502 on failure.
- granary.source.creation_result(content=None, description=None, abort=False, error_plain=None, error_html=None)[source]
Creates a new
CreationResult
.
- class granary.source.SourceMeta(name, bases, class_dict)[source]
Bases:
type
Source metaclass. Registers all source classes in the sources global.
- class granary.source.Source[source]
Bases:
object
Abstract base class for a source (e.g. Facebook, Twitter).
Concrete subclasses must override the class constants below and implement
get_activities()
.- EMBED_POST
the HTML for embedding a post. Should have a
%(url)s
placeholder for the post URL and optionally a%(content)s
placeholder for the post content.- Type:
- HTML2TEXT_OPTIONS
maps str html2text option names to values. https://github.com/Alir3z4/html2text/blob/master/docs/usage.md#available-options
- Type:
- TRUNCATE_TEXT_LENGTH
optional character limit to truncate to. Defaults to Twitter’s limit, 280 characters as of 2019-10-12.
- Type:
- TRUNCATE_URL_LENGTH
optional number of characters that URLs count for. Defaults to Twitter’s, 23 as of 2019-10-12.
- Type:
- OPTIMIZED_COMMENTS
whether
get_comment()
is optimized and only fetches the requested comment. If False,get_comment()
fetches many or all of the post’s comments to find the requested one.- Type:
- get_actor(user_id=None)[source]
Fetches and returns a user.
- Parameters:
user_id – str, defaults to current user
- Returns:
ActivityStreams actor
- Return type:
- get_activities(*args, **kwargs)[source]
Fetches and returns a list of activities.
See
get_activities_response()
for args and kwargs.
- get_activities_response(user_id=None, group_id=None, app_id=None, activity_id=None, start_index=0, count=0, etag=None, min_id=None, cache=None, fetch_replies=False, fetch_likes=False, fetch_shares=False, include_shares=True, fetch_events=False, fetch_mentions=False, search_query=None, scrape=False, **kwargs)[source]
Fetches and returns ActivityStreams activities and response details.
Subclasses should override this. See
get_activities()
for an alternative that just returns the list of activities.If user_id is provided, only that user’s activity(s) are included. start_index and count determine paging, as described in the spec: http://activitystrea.ms/draft-spec.html#anchor14
app id is just object id: http://opensocial-resources.googlecode.com/svn/spec/2.0/Social-Data.xml#appId
group id is string id of group or @self, @friends, @all, @search: http://opensocial-resources.googlecode.com/svn/spec/2.0/Social-Data.xml#Group-ID
The
fetch_*
kwargs all default to False because they often require extra API round trips. Some sources return replies, likes, and shares in the same initial call, so they may be included even if you don’t set their kwarg to True.- Parameters:
user_id (str) – defaults to the currently authenticated user
group_id (str) – one of
@self
,@all
,@friends
,@search
. defaults to@friends
app_id (str) –
activity_id (str) –
start_index (int) – >= 0
count (int) – >= 0
etag (str) – optional ETag to send with the API request. Results will only be returned if the ETag has changed. Should include enclosing double quotes, e.g.
"ABC123"
min_id (only) – return activities with ids greater than this
cache (dict) – optional, used to cache metadata like comment and like counts per activity across calls. Used to skip expensive API calls that haven’t changed.
fetch_replies (bool) – whether to fetch each activity’s replies also
fetch_likes (bool) – whether to fetch each activity’s likes also
include_shares (bool) – whether to include share activities
fetch_shares (bool) – whether to fetch each activity’s shares also
fetch_events (bool) – whether to fetch the user’s events also
fetch_mentions (bool) – whether to fetch posts that mention the user
search_query (str) – an optional search query, only for use with @search group_id
scrape (bool) – whether to scrape activities from HTML (etc) instead of using an API. Not supported by all sources.
kwargs – some subclasses accept extra kwargs. See their docs for details.
- Returns:
Response values based on OpenSocial ActivityStreams REST API.
The returned dict has at least these keys:
items
(list of dict): activitiesstartIndex
(int or None)itemsPerPage
(int)totalResults
(int or None, eg if it can’t be calculated efficiently)filtered
: Falsesorted
: FalseupdatedSince
: Falseetag
(str): ETag returned by the API’s initial call to get activities
- Return type:
- Raises:
ValueError – if any argument is invalid for this source
NotImplementedError – if the source doesn’t support the requested operation, eg Facebook doesn’t support search.
- classmethod make_activities_base_response(activities, *args, **kwargs)[source]
Generates a base response dict for
get_activities_response()
.See
get_activities()
for args and kwargs.
- scraped_to_activities(scraped, count=None, fetch_extras=False, cookie=None)[source]
Converts scraped HTML (or JSON, etc) to AS activities.
Used for scraping data from the web instead of using an API. Useful for sources with APIs that are restricted or have difficult approval processes.
- Parameters:
- Returns:
([AS activities], AS logged in actor (ie viewer))
- Return type:
- scraped_to_activity(scraped)[source]
Converts scraped HTML (or JSON, etc) to a single AS activity.
Used for scraping data from the web instead of using an API. Useful for sources with APIs that are restricted or have difficult approval processes.
- scraped_to_actor(scraped)[source]
Converts HTML from a profile page to an AS1 actor.
- Args
html (str): HTML from a profile page
- Returns:
AS1 actor
- Return type:
- merge_scraped_reactions(scraped, activity)[source]
Converts and merges scraped likes and reactions into an activity.
New likes and emoji reactions are added to the activity in
tags
. Existing likes and emoji reactions intags
are ignored.
- create(obj, include_link='omit', ignore_formatting=False)[source]
Creates a new object: a post, comment, like, share, or RSVP.
Subclasses should override this. Different sites will support different functionality; check each subclass for details. The actor will usually be the authenticated user.
- Parameters:
obj (dict) – ActivityStreams object. At minimum, must have the content field. objectType is strongly recommended.
include_link (str) –
INCLUDE_LINK
,OMIT_LINK
, orINCLUDE_IF_TRUNCATED
; whether to include a link to the object (if it has one) in the content.ignore_formatting (bool) – whether to use content text as is, instead of converting its HTML to plain text styling (newlines, etc.)
- Returns:
The result.
content
will be a dict or None. If the newly created object has an id or permalink, they’ll be provided in the values forid
andurl
.- Return type:
- preview_create(obj, include_link='omit', ignore_formatting=False)[source]
Previews creating a new object: a post, comment, like, share, or RSVP.
Returns HTML that previews what
create()
with the same object will do.Subclasses should override this. Different sites will support different functionality; check each subclass for details. The actor will usually be the authenticated user.
- Parameters:
obj (dict) – ActivityStreams object. At minimum, must have the
content
field.objectType
is strongly recommended.include_link (str) –
INCLUDE_LINK
,OMIT_LINK
, orINCLUDE_IF_TRUNCATED
; whether to include a link to the object (if it has one) in the content.ignore_formatting (bool) – whether to use content text as is, instead of converting its HTML to plain text styling (newlines, etc.)
- Returns:
The result. content will be a dict or
None
.- Return type:
- delete(id)[source]
Deletes a post.
Generally only supports posts that were authored by the authenticating user.
- Parameters:
id (str) – silo object id
- Return type:
- preview_delete(id)[source]
Previews deleting a post.
- Parameters:
id (str) – silo object id
- Return type:
- get_comment(comment_id, activity_id=None, activity_author_id=None, activity=None)[source]
Fetches and returns a comment.
Subclasses should override this.
- Parameters:
- Returns:
ActivityStreams comment object
- Return type:
- Raises:
ValueError – if any argument is invalid for this source
- get_like(activity_user_id, activity_id, like_user_id, activity=None)[source]
Fetches and returns a ‘like’.
Default implementation that fetches the activity and its likes, then searches for a like by the given user. Subclasses should override this if they can optimize the process.
- Parameters:
- Returns:
ActivityStreams like activity
- Return type:
- get_reaction(activity_user_id, activity_id, reaction_user_id, reaction_id, activity=None)[source]
Fetches and returns a reaction.
Default implementation that fetches the activity and its reactions, then searches for this specific reaction. Subclasses should override this if they can optimize the process.
- Parameters:
- Returns:
ActivityStreams reaction activity
- Return type:
Fetches and returns a share.
- Parameters:
- Returns:
an ActivityStreams share activity
- Return type:
- get_rsvp(activity_user_id, event_id, user_id, event=None)[source]
Fetches and returns an RSVP.
- Parameters:
Returns: dict, an ActivityStreams RSVP activity object
- get_blocklist()[source]
Fetches and returns the current user’s block list.
…ie the users that the current user is blocking. The exact semantics of blocking vary from silo to silo.
- Returns:
actor objects
- Return type:
sequence of dict
- get_blocklist_ids()[source]
Returns the current user’s block list as a list of silo-specific user ids.
- user_to_actor(user)[source]
Converts a user to an actor.
The returned object will have at least a
url
field. If the user has multiple URLs, there will also be aurls
list field whose elements are dicts withvalue
URL.
- static postprocess_activity(activity)[source]
Does source-independent post-processing of an activity, in place.
Right now just populates the
title
field.- Parameters:
activity (dict) –
- static postprocess_object(obj)[source]
Does source-independent post-processing of an object, in place.
Populates
location.position
based on latitude and longitude.- Parameters:
object (dict) –
- base_object(obj)[source]
Returns the
base
silo object that an object operates on.For example, if the object is a comment, this returns the post that it’s a comment on. If it’s an RSVP, this returns the event. The id in the returned object is silo-specific, ie not a tag URI.
Subclasses may override this.
- classmethod base_id(url)[source]
Guesses the id of the object in the given URL.
- Returns:
str or None
- truncate(content, url, include_link, type=None, quote_url=None)[source]
Shorten text content to fit within a character limit.
Character limit and URL character length are taken from
TRUNCATE_TEXT_LENGTH
andTRUNCATE_URL_LENGTH
.- Parameters:
- Returns:
the possibly shortened and ellipsized text
- Return type:
twitter
Twitter source class.
Uses the v1.1 REST API: https://developer.twitter.com/en/docs/api-reference-index
The Audience Targeting to
field is set to @public
or @private
based
on whether the tweet author’s protected
field is true or false.
https://dev.twitter.com/docs/platform-objects/users
- class granary.twitter.OffsetTzinfo(utc_offset=0)[source]
Bases:
tzinfo
A simple, DST-unaware tzinfo from given utc offset in seconds.
- __init__(utc_offset=0)[source]
Constructor.
- Parameters:
utc_offset – Offset of time zone from UTC in seconds
- class granary.twitter.Twitter(access_token_key, access_token_secret, username=None, scrape_headers=None)[source]
Bases:
Source
Twitter source class. See file docstring and
Source
for details.- __init__(access_token_key, access_token_secret, username=None, scrape_headers=None)[source]
Constructor.
Twitter now requires authentication in v1.1 of their API. You can get an OAuth access token by creating an app here: https://dev.twitter.com/apps/new
- get_actor(screen_name=None)[source]
Returns a user as a JSON ActivityStreams actor dict.
- Parameters:
screen_name (str) – username. Defaults to the current user.
- get_activities_response(user_id=None, group_id=None, app_id=None, activity_id=None, start_index=0, count=0, etag=None, min_id=None, cache=None, fetch_replies=False, fetch_likes=False, fetch_shares=False, include_shares=True, fetch_events=False, fetch_mentions=False, search_query=None, scrape=False, **kwargs)[source]
Fetches posts and converts them to ActivityStreams activities.
See
source.Source.get_activities_response()
for details.app_id
is ignored.min_id
is translated to Twitter’ssince_id
.The code for handling ETags (and 304 Not Changed responses and setting
If-None-Match
) is here, but unused right now since Twitter evidently doesn’t support ETags. From https://dev.twitter.com/discussions/5800 : “I’ve confirmed with our team that we’re not explicitly supporting this family of features.”Likes (nee favorites) are scraped from twitter.com, since Twitter’s REST API doesn’t offer a way to fetch them. You can also get them from the Streaming API, though, and convert them with
streaming_event_to_object()
. https://dev.twitter.com/docs/streaming-apis/messages#Events_eventShares (ie retweets) are fetched with a separate API call per tweet: https://dev.twitter.com/docs/api/1.1/get/statuses/retweets/%3Aid
However, retweets are only fetched for the first 15 tweets that have them, since that’s Twitter’s rate limit per 15 minute window. :( https://dev.twitter.com/docs/rate-limiting/1.1/limits
Quote tweets are fetched by searching for the possibly quoted tweet’s ID, using the OR operator to search up to 5 IDs at a time, and then checking the
quoted_status_id_str
field: https://dev.twitter.com/overview/api/tweets#quoted_status_id_strUse the group_id @self to retrieve a user_id’s timeline. If
user_id
is None or@me
, it will return tweets for the current API user.group_id can be used to specify the slug of a list for which to return tweets. By default the current API user’s lists will be used, but lists owned by other users can be fetched by explicitly passing a username to
user_id
, e.g. to fetch tweets from the list@exampleuser/example-list
you would callget_activities(user_id='exampleuser', group_id='example-list')
.Twitter replies default to including a mention of the user they’re replying to, which overloads mentions a bit. When fetch_mentions is True, we determine that a tweet mentions the current user if it @-mentions their username and:
it’s not a reply, OR
it’s a reply, but not to the current user, AND
the tweet it’s replying to doesn’t @-mention the current user
- Raises:
NotImplementedError – if
fetch_likes
is True butscrape_headers
was not provided to the constructor.
XXX HACK: this is currently hacked for Bridgy to NOT pass
min_id
to the request for fetching activity tweets themselves, but to pass it to all of the requests for filling in replies, retweets, etc. That’s because we want to find new replies and retweets of older initial tweets. TODO: find a better way.
- fetch_replies(activities, min_id=None)[source]
Fetches and injects Twitter replies into a list of activities, in place.
Includes indirect replies ie reply chains, not just direct replies. Searches for @-mentions, matches them to the original tweets with
in_reply_to_status_id_str
, and recurses until it’s walked the entire tree.
- fetch_mentions(username, tweets, min_id=None)[source]
Fetches a user’s @-mentions and returns them as ActivityStreams.
Tries to only include explicit mentions, not mentions automatically created by @-replying. See
get_activities_response()
for details.
- get_comment(comment_id, activity_id=None, activity_author_id=None, activity=None)[source]
Returns an ActivityStreams comment object.
Returns an ActivityStreams ‘share’ activity object.
- get_blocklist()[source]
Returns the current user’s block list.
May make multiple API calls, using cursors, to fully fetch large blocklists. https://dev.twitter.com/overview/api/cursoring
Block lists may have up to 10k users, but each API call only returns 100 at most, and the API endpoint is rate limited to 15 calls per user per 15m. So if a user has >1500 users on their block list, we can’t get the whole thing at once. :(
- Returns:
actors
- Return type:
- Raises:
source.RateLimited – if we hit the rate limit. The partial attribute will
have the list of user ids we fetched before hitting the limit. –
- get_blocklist_ids()[source]
Returns the current user’s block list as a list of Twitter user ids.
May make multiple API calls, using cursors, to fully fetch large blocklists. https://dev.twitter.com/overview/api/cursoring
Subject to the same rate limiting as
get_blocklist()
, but each API call returns ~4k ids, so realistically this can actually fetch blocklists of up to 75k users at once. Beware though, many Twitter users have even more!- Returns:
Twitter user ids
- Return type:
sequence of str
- Raises:
source.RateLimited – if we hit the rate limit. The partial attribute will
have the list of user ids we fetched before hitting the limit. –
- create(obj, include_link='omit', ignore_formatting=False)[source]
Creates a tweet, reply tweet, retweet, or favorite.
- Parameters:
- Returns:
content will be a dict with
id
,url
, andtype
keys (all optional) for the newly created Twitter object (or None)- Return type:
- preview_create(obj, include_link='omit', ignore_formatting=False)[source]
Previews creating a tweet, reply tweet, retweet, or favorite.
- Parameters:
- Returns:
content will be an HTML snippet
- Return type:
CreationResult or None
- upload_images(images)[source]
Uploads one or more images from web URLs.
https://dev.twitter.com/rest/reference/post/media/upload
Note that files and JSON bodies in media POST API requests are not included in OAuth signatures. https://developer.twitter.com/en/docs/media/upload-media/uploading-media/media-best-practices
- upload_video(url)[source]
Uploads a video from web URLs using the chunked upload process.
Chunked upload consists of multiple API calls:
command=INIT
, which allocates the media idcommand=APPEND
for each 5MB block, up to 15MB totalcommand=FINALIZE
https://developer.twitter.com/en/docs/media/upload-media/uploading-media/chunked-media-upload
- Parameters:
url (str) – URL of images
- Returns:
media id
- Return type:
str, or
CreationResult
on error
- delete(id)[source]
Deletes a tweet. The authenticated user must have authored it.
- Parameters:
- Returns:
content is Twitter API response dict
- Return type:
- urlopen(url, parse_response=True, **kwargs)[source]
Wraps
urllib.request.urlopen()
and adds an OAuth signature.
- base_object(obj)[source]
Returns the “base” silo object that an object operates on.
Includes special handling for Twitter photo and video URLs, eg:
https://twitter.com/nelson/status/447465082327298048/photo/1
https://twitter.com/nelson/status/447465082327298048/video/1
- tweet_to_object(tweet)[source]
Converts a tweet to an object.
- Parameters:
tweet (dict) – a decoded JSON tweet
- Returns:
an ActivityStreams object dict, ready to be JSON-encoded
- streaming_event_to_object(event)[source]
Converts a Streaming API event to an object.
https://dev.twitter.com/docs/streaming-apis/messages#Events_event
Right now, only converts favorite events to like objects.
- static rfc2822_to_iso8601(time_str)[source]
Converts a timestamp string from RFC 2822 format to ISO 8601.
- Example RFC 2822 timestamp string generated by Twitter:
Wed May 23 06:01:13 +0000 2007
- Resulting ISO 8610 timestamp string:
2007-05-23T06:01:13