F# Data Toolbox


F# Data Toolbox: Twitter type provider

The Twitter type provider makes Twitter data easily accessible by providing a light wrapper around the Twitter API.

Connecting to Twitter

Twitter requires developers to register their applications to gain access to its API. You have to register your application at Twitter Apps. After registration, Twitter provides API key and API secret to authenticate the application.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
// First, reference the locations where F# Data and 
// F# Data Toolbox are located (using '#I' is required here!)
#I @"packages/FSharp.Data.Toolbox.Twitter.0.3/lib/net40"
#I @"packages/FSharp.Data.2.1.1/lib/net40"

// The Twitter reference needs to come before FSharp.Data.dll
// (see the big warning box below for more!)
#r "FSharp.Data.Toolbox.Twitter.dll"
#r "FSharp.Data.dll"
open FSharp.Data.Toolbox.Twitter

let key = "mKQL29XNemjQbLlQ8t0pBg"
let secret = "T27HLDve1lumQykBUgYAbcEkbDrjBe6gwbu0gqi4saM"

WARNING: Unfortunately, F# Interactive is quite sensitive to how you reference the packages when using F# Data Toolbox. To make the Twitter type provider work correctly in F# Interactive, you need to:

  • Use the #I directive to reference the path where the two libraries are located (rather than usign #r with a full relative path)

  • Reference FSharp.Data.Toolbox.Twitter.dll before referencing FSharp.Data.dll as on the first two lines above.

  • If you are using the Twitter provider in a compiled project, you will also need to add reference to System.Windows.Forms.

This second point is required so that the JSON type provider (used in F# Data Toolbox) can locate sample JSON files from the embedded metadata of the FSharp.Data.Toolbox.Twitter.dll assembly. An alternative is to copy the sample JSON files together with the assembly.

There are two types of possible connections to Twitter, application-only and full OAuth authentication. They provide different access rights and different number of allowed requests per time window.

Connecting with application-only authentication

The application-only authentication provides access that's limited to data reachable without the full user context. For example, it allows accessing friends and followers and searching in tweets. This is how the application obtains acess credentials:

1: 
let twitter = Twitter.AuthenticateAppOnly(key, secret)

Connecting with OAuth

This method of access provides full user context for the application. Compared to application-only access, it can also access Streaming, search for users and post tweets on behalf of the user.

To connect with OAuth, we first create a Twitter connector. with your user name and password. You'll get a PIN that you use as an argument for the Connect function. This user authentication allows full access to Twitter APIs.

1: 
2: 
3: 
4: 
let connector = Twitter.Authenticate(key, secret) 

// Run this part after you obtain PIN
let twitter = connector.Connect("7808652")

Twitter login window

After connecting to Twitter, you can call methods to access Twitter users, list followers and friends, search for tweets and access the global Twitter stream. All the methods send http requests to Twitter and return JSON documents. They are parsed using JSON type provider, which allows to access individual properties.

Accessing friends and followers

The following examples show how to access lists of followers and friends (followed accounts). Users can be identified either by their Twitter name, or by their user ID number.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
// Get a list of ID numbers of friends and followers 
// for the current signed-in user
// (requires full authentication)
let friends = twitter.Connections.FriendsIds()
let followers = twitter.Connections.FollowerIds()
printfn "Number of friends: %d" (friends.Ids |> Seq.length)
printfn "Number of followers: %d" (followers.Ids |> Seq.length)

// Get a list IDs of friends and followers for a specific user 
let followersFSorg = twitter.Connections.FriendsIds(userId=880772426L)
let friendsFSorg = twitter.Connections.FollowerIds(screenName="fsharporg")

// Get information about connection between specific users
let fs = twitter.Connections.Friendship(880772426L, 94144339L)
fs.Relationship.Source.ScreenName
fs.Relationship.Target.ScreenName
fs.Relationship.Source.Following
fs.Relationship.Source.FollowedBy

We can also search for information about a list of users, specified either by IDs or by their screen names. It's possible to search for up to 100 users at a time.

1: 
2: 
3: 
let friendInfos = twitter.Users.Lookup(friends.Ids |> Seq.truncate 100)
for friend in friendInfos do
  printfn "%s (@%s)\t\t%d" friend.Name friend.ScreenName friend.Id

Searching for tweets

We can search Twitter for tweets using keywords. The following snippet shows how to search for tweets containing the #fsharp tag.

1: 
2: 
3: 
4: 
5: 
let fsharpTweets = twitter.Search.Tweets("#fsharp", count=100)

for status in fsharpTweets.Statuses do
  printfn "@%s: %s" status.User.ScreenName status.Text
  

Accessing timelines

Let's look at how to access Twitter timelines. Timelines show a stream of tweets from followed accounts.

We can access timelines for specific users, or the home timeline of the current signed-in user.

1: 
2: 
3: 
4: 
5: 
6: 
// Access home timeline
// (requires full user authentication)
let home = twitter.Timelines.HomeTimeline()

// Timeline of a specific user, up to a specified number of tweets
let timeline = twitter.Timelines.Timeline("fsharporg", 10)

We can display the Timeline in a web browser. We first create a web browser window. Then we download timeline for a specific user, in this case it's @fsharporg. Finally, we display individual tweets in the web browser.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
23: 
open System.Windows.Forms
open FSharp.Control
open FSharp.WebBrowser

// Create a windonw wtih web browser
let frm = new Form(TopMost = true, Visible = true, Width = 500, Height = 400)
let btn = new Button(Text = "Pause", Dock = DockStyle.Top)
let web = new WebBrowser(Dock = DockStyle.Fill)
frm.Controls.Add(web)
frm.Controls.Add(btn)
web.Output.StartList()

// Display timeline
let timeline = twitter.Timelines.Timeline("fsharporg")
for tweet in timeline do
    web.Output.AddItem "<strong>%s</strong>: %s" tweet.User.Name tweet.Text


// Access mentions timeline
// (requires full user authentication)
let mention = twitter.Timelines.MentionTimeline()
for tweet in mention do
    web.Output.AddItem "<strong>%s</strong>: %s" tweet.User.Name tweet.Text

Twitter timeline

Streaming data

Streaming allows access to live Twitter data, as they're posted. To access Streaming API, the application must have full user authentication.

If we reuse the web browser window created in the previous code sample, we can display a random sample of tweets in the following way.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
// Display stream with live data
web.Output.StartList()

let sample = twitter.Streaming.SampleTweets()
sample.TweetReceived |> Observable.guiSubscribe (fun status ->
    match status.Text, status.User with
    | Some text, Some user ->
        web.Output.AddItem "<strong>%s</strong>: %s" user.Name text
    | _ -> ()  )
sample.Start()

sample.Stop()

We can also search the Twitter stream for a specific hashtag or phrase. The following code will filter all tweets that contain the word "fsharp" from the global stream of tweets.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
// Display live search data 
web.Output.StartList()

let search = twitter.Streaming.FilterTweets ["fsharp"]
search.TweetReceived |> Observable.guiSubscribe (fun status ->
    match status.Text, status.User with
    | Some text, Some user ->
        web.Output.AddItem "<strong>%s</strong>: %s" user.Name text
    | _ -> ()  )
search.Start()

search.Stop()
Multiple items
namespace FSharp

--------------------
namespace Microsoft.FSharp
Multiple items
namespace FSharp.Data

--------------------
namespace Microsoft.FSharp.Data
namespace FSharp.Data.Toolbox
namespace FSharp.Data.Toolbox.Twitter
val key : string

Full name: TwitterProvider.key
val secret : string

Full name: TwitterProvider.secret
val twitter : Twitter

Full name: TwitterProvider.twitter
Multiple items
type Twitter =
  new : context:TwitterContext -> Twitter
  member RequestRawData : url:string * query:(string * string) list -> string
  member Connections : Connections
  member Search : Search
  member Streaming : Streaming
  member Timelines : Timelines
  member Trends : Trends
  member Tweets : Tweet
  member Users : Users
  static member Authenticate : consumer_key:string * consumer_secret:string -> TwitterConnector
  ...

Full name: FSharp.Data.Toolbox.Twitter.Twitter

--------------------
new : context:TwitterContext -> Twitter
static member Twitter.AuthenticateAppOnly : consumer_key:string * consumer_secret:string -> Twitter
val connector : TwitterConnector

Full name: TwitterProvider.connector
static member Twitter.Authenticate : consumer_key:string * consumer_secret:string -> TwitterConnector
abstract member TwitterConnector.Connect : string -> Twitter
val friends : FSharp.Data.JsonProvider<...>.Root

Full name: TwitterProvider.friends
property Twitter.Connections: Connections
member Connections.FriendsIds : ?userId:int64 * ?screenName:string * ?cursor:int64 * ?count:int -> FSharp.Data.JsonProvider<...>.Root
val followers : FSharp.Data.JsonProvider<...>.Root

Full name: TwitterProvider.followers
member Connections.FollowerIds : ?userId:int64 * ?screenName:string * ?cursor:int64 * ?count:int -> FSharp.Data.JsonProvider<...>.Root
val printfn : format:Printf.TextWriterFormat<'T> -> 'T

Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.printfn
property FSharp.Data.JsonProvider<...>.Root.Ids: int64 []
module Seq

from Microsoft.FSharp.Collections
val length : source:seq<'T> -> int

Full name: Microsoft.FSharp.Collections.Seq.length
val followersFSorg : FSharp.Data.JsonProvider<...>.Root

Full name: TwitterProvider.followersFSorg
val friendsFSorg : FSharp.Data.JsonProvider<...>.Root

Full name: TwitterProvider.friendsFSorg
val fs : FSharp.Data.JsonProvider<...>.Root

Full name: TwitterProvider.fs
member Connections.Friendship : ?sourceId:int64 * ?targetId:int64 -> FSharp.Data.JsonProvider<...>.Root
property FSharp.Data.JsonProvider<...>.Root.Relationship: FSharp.Data.JsonProvider<...>.Relationship
property FSharp.Data.JsonProvider<...>.Relationship.Source: FSharp.Data.JsonProvider<...>.Source
property FSharp.Data.JsonProvider<...>.Source.ScreenName: string
property FSharp.Data.JsonProvider<...>.Relationship.Target: FSharp.Data.JsonProvider<...>.Target
property FSharp.Data.JsonProvider<...>.Target.ScreenName: string
property FSharp.Data.JsonProvider<...>.Source.Following: bool
property FSharp.Data.JsonProvider<...>.Source.FollowedBy: bool
val friendInfos : FSharp.Data.JsonProvider<...>.Root []

Full name: TwitterProvider.friendInfos
property Twitter.Users: Users
member Users.Lookup : screenNames:seq<string> -> FSharp.Data.JsonProvider<...>.Root []
member Users.Lookup : userIds:seq<int64> -> FSharp.Data.JsonProvider<...>.Root []
val truncate : count:int -> source:seq<'T> -> seq<'T>

Full name: Microsoft.FSharp.Collections.Seq.truncate
val friend : FSharp.Data.JsonProvider<...>.Root
property FSharp.Data.JsonProvider<...>.Root.Name: string
property FSharp.Data.JsonProvider<...>.Root.ScreenName: string
property FSharp.Data.JsonProvider<...>.Root.Id: int64
val fsharpTweets : FSharp.Data.JsonProvider<...>.Root

Full name: TwitterProvider.fsharpTweets
property Twitter.Search: Search
member Search.Tweets : query:string * ?lang:string * ?geocode:string * ?locale:string * ?count:int * ?sinceId:int64 * ?maxId:int64 * ?until:string -> FSharp.Data.JsonProvider<...>.Root
val status : FSharp.Data.JsonProvider<...>.Status
property FSharp.Data.JsonProvider<...>.Root.Statuses: FSharp.Data.JsonProvider<...>.Status []
property FSharp.Data.JsonProvider<...>.Status.User: FSharp.Data.JsonProvider<...>.User
property FSharp.Data.JsonProvider<...>.User.ScreenName: string
property FSharp.Data.JsonProvider<...>.Status.Text: string
val home : FSharp.Data.JsonProvider<...>.Root []

Full name: TwitterProvider.home
property Twitter.Timelines: Timelines
member Timelines.HomeTimeline : ?count:int * ?sinceId:int64 * ?maxId:int64 * ?trimUser:bool * ?contributorDetails:bool * ?includeEntities:bool -> FSharp.Data.JsonProvider<...>.Root []
val timeline : FSharp.Data.JsonProvider<...>.Root []

Full name: TwitterProvider.timeline
member Timelines.Timeline : screenName:string * ?count:int * ?maxId:int64 -> FSharp.Data.JsonProvider<...>.Root []
member Timelines.Timeline : userId:int64 * ?count:int * ?maxId:int64 -> FSharp.Data.JsonProvider<...>.Root []
namespace System
namespace System.Windows
namespace System.Windows.Forms
Multiple items
namespace FSharp.Control

--------------------
namespace Microsoft.FSharp.Control
namespace FSharp.WebBrowser
val frm : Form

Full name: TwitterProvider.frm
Multiple items
type Form =
  inherit ContainerControl
  new : unit -> Form
  member AcceptButton : IButtonControl with get, set
  member Activate : unit -> unit
  member ActiveMdiChild : Form
  member AddOwnedForm : ownedForm:Form -> unit
  member AllowTransparency : bool with get, set
  member AutoScale : bool with get, set
  member AutoScaleBaseSize : Size with get, set
  member AutoScroll : bool with get, set
  member AutoSize : bool with get, set
  ...
  nested type ControlCollection

Full name: System.Windows.Forms.Form

--------------------
Form() : unit
val btn : Button

Full name: TwitterProvider.btn
Multiple items
type Button =
  inherit ButtonBase
  new : unit -> Button
  member AutoSizeMode : AutoSizeMode with get, set
  member DialogResult : DialogResult with get, set
  member NotifyDefault : value:bool -> unit
  member PerformClick : unit -> unit
  member ToString : unit -> string
  event DoubleClick : EventHandler
  event MouseDoubleClick : MouseEventHandler

Full name: System.Windows.Forms.Button

--------------------
Button() : unit
type DockStyle =
  | None = 0
  | Top = 1
  | Bottom = 2
  | Left = 3
  | Right = 4
  | Fill = 5

Full name: System.Windows.Forms.DockStyle
field DockStyle.Top = 1
val web : WebBrowser

Full name: TwitterProvider.web
Multiple items
type WebBrowser =
  inherit WebBrowserBase
  new : unit -> WebBrowser
  member AllowNavigation : bool with get, set
  member AllowWebBrowserDrop : bool with get, set
  member CanGoBack : bool
  member CanGoForward : bool
  member Document : HtmlDocument
  member DocumentStream : Stream with get, set
  member DocumentText : string with get, set
  member DocumentTitle : string
  member DocumentType : string
  ...

Full name: System.Windows.Forms.WebBrowser

--------------------
WebBrowser() : unit
field DockStyle.Fill = 5
property Control.Controls: Control.ControlCollection
Control.ControlCollection.Add(value: Control) : unit
property WebBrowser.Output: WebBrowserOutput
member WebBrowserOutput.StartList : unit -> unit
member WebBrowserOutput.StartList : f:Printf.StringFormat<'a0,unit> -> 'a0
val tweet : FSharp.Data.JsonProvider<...>.Root
member WebBrowserOutput.AddItem : f:Printf.StringFormat<'a,unit> -> 'a
property FSharp.Data.JsonProvider<...>.Root.User: FSharp.Data.JsonProvider<...>.User
property FSharp.Data.JsonProvider<...>.User.Name: string
property FSharp.Data.JsonProvider<...>.Root.Text: string
val mention : FSharp.Data.JsonProvider<...>.Root []

Full name: TwitterProvider.mention
member Timelines.MentionTimeline : ?count:int * ?sinceId:int64 * ?maxId:int64 * ?trimUser:bool * ?contributorDetails:bool * ?includeEntities:bool -> FSharp.Data.JsonProvider<...>.Root []
val sample : TwitterStream<FSharp.Data.JsonProvider<...>.Root>

Full name: TwitterProvider.sample
property Twitter.Streaming: Streaming
member Streaming.SampleTweets : unit -> TwitterStream<FSharp.Data.JsonProvider<...>.Root>
property TwitterStream.TweetReceived: IEvent<FSharp.Data.JsonProvider<...>.Root>
Multiple items
module Observable

from FSharp.Control

--------------------
module Observable

from Microsoft.FSharp.Control
val guiSubscribe : f:('a -> unit) -> obs:System.IObservable<'a> -> System.IDisposable

Full name: FSharp.Control.Observable.guiSubscribe
val status : FSharp.Data.JsonProvider<...>.Root
property FSharp.Data.JsonProvider<...>.Root.Text: Option<string>
property FSharp.Data.JsonProvider<...>.Root.User: Option<FSharp.Data.JsonProvider<...>.User>
union case Option.Some: Value: 'T -> Option<'T>
val text : string
val user : FSharp.Data.JsonProvider<...>.User
abstract member TwitterStream.Start : unit -> unit
abstract member TwitterStream.Stop : unit -> unit
val search : TwitterStream<FSharp.Data.JsonProvider<...>.Root>

Full name: TwitterProvider.search
member Streaming.FilterTweets : keywords:seq<string> -> TwitterStream<FSharp.Data.JsonProvider<...>.Root>
Fork me on GitHub