CSV Type Provider

This article demonstrates how to use the CSV type provider to read CSV files in a statically typed way.

The CSV type provider takes a sample CSV as input and generates a type based on the data present on the columns of that sample. The column names are obtained from the first (header) row, and the types are inferred from the values present on the subsequent rows.

Introducing the provider

The type provider is located in the FSharp.Data.dll assembly. Assuming the package is referenged we can access its namespace as follows:

open FSharp.Data

Parsing stock prices

The Yahoo Finance web site provides daily stock prices in a CSV format that has the following structure (you can find a larger example in the data/MSFT.csv file):

Date,Open,High,Low,Close,Volume,Adj Close

As usual with CSV files, the first row contains the headers (names of individual columns) and the next rows define the data. We can pass reference to the file to CsvProvider to get a strongly typed view of the file:

type Stocks = CsvProvider<"../data/MSFT.csv", ResolutionFolder=__SOURCE_DIRECTORY__>

The generated type provides two static methods for loading data. The Parse method can be used if we have the data in a string value. The Load method allows reading the data from a file or from a web resource (and there's also an asynchronous AsyncLoad version). We could also have used a web URL instead of a local file in the sample parameter of the type provider. The following sample calls the Load method with an URL that points to a live CSV file on the Yahoo finance web site:

// Download the stock prices
let msft = Stocks.Load(__SOURCE_DIRECTORY__ + "/../data/MSFT.csv").Cache()

// Look at the most recent row. Note the 'Date' property
// is of type 'DateTime' and 'Open' has a type 'decimal'
let firstRow = msft.Rows |> Seq.head
let lastDate = firstRow.Date
let lastOpen = firstRow.Open

// Print the prices in the HLOC format
for row in msft.Rows do
  printfn "HLOC: (%A, %A, %A, %A)" row.High row.Low row.Open row.Close

The generated type has a property Rows that returns the data from the CSV file as a collection of rows. We iterate over the rows using a for loop. As you can see the (generated) type for rows has properties such as High, Low and Close that correspond to the columns in the CSV file.

As you can see, the type provider also infers types of individual rows. The Date property is inferred to be a DateTime (because the values in the sample file can all be parsed as dates) while HLOC prices are inferred as decimal.

Charting stock prices

We can use the XPlot.Plotly library to draw a simple line chart showing how the price of MSFT stocks changes:

// Load the XPlot.Plotly library
#r "nuget: XPlot.Plotly, Version=3.0.1"
open XPlot.Plotly
open System
// Visualize the stock prices
[ for row in msft.Rows -> row.Date, row.Open ]
|> Chart.Line
No value returned by any evaluator

As one more example, we use the Candlestick chart to get a more detailed look at the data over the last month:

// Get last months' prices in HLOC format
let recent =
  [ for row in msft.Rows do
      if row.Date > DateTime.Parse("9 Sep 2017") then
        yield row.Date, row.High, row.Low, row.Open, row.Close ]
// Visualize prices using Candlestick chart
No value returned by any evaluator

Using units of measure

Another interesting feature of the CSV type provider is that it supports F# units of measure. If the header includes the name or symbol of one of the standard SI units, then the generated type returns values annotated with the appropriate unit.

In this section, we use a simple file data/SmallTest.csv which looks as follows:

Name,  Distance (metre), Time (s)
First, 50.0,             3.7

As you can see, the second and third columns are annotated with metre and s, respectively. To use units of measure in our code, we need to open the namespace with standard unit names. Then we pass the SmallTest.csv file to the type provider as a static argument. Also note that in this case we're using the same data at runtime, so we use the GetSample method instead of calling Load and passing the same parameter again.

let small = CsvProvider<"../data/SmallTest.csv", ResolutionFolder=__SOURCE_DIRECTORY__>.GetSample()

We can also use the default constructor instead of the GetSample static method:

let small2 = new CsvProvider<"../data/SmallTest.csv", ResolutionFolder=__SOURCE_DIRECTORY__>()

but the VisualStudio IntelliSense for the type provider parameters doesn't work when we use a default constructor for a type provider, so we'll keep using GetSample instead.

As in the previous example, the small value exposes the rows using the Rows property. The generated properties Distance and Time are now annotated with units. Look at the following simple calculation:

open FSharp.Data.UnitSystems.SI.UnitNames

for row in small.Rows do
  let speed = row.Distance / row.Time
  if speed > 15.0M<metre/second> then
    printfn "%s (%A m/s)" row.Name speed

The numerical values of Distance and Time are both inferred as decimal (because they are small enough). Thus the type of speed becomes decimal<metre/second>. The compiler can then statically check that we're not comparing incompatible values - e.g. number in meters per second against a value in kilometres per hour.

Custom separators and tab-separated files

By default, the CSV type provider uses comma (,) as a separator. However, CSV files sometime use a different separator character than ,. In some European countries, , is already used as the numeric decimal separator, so a semicolon (;) is used instead to separate CSV columns. The CsvProvider has an optional Separator static parameter where you can specify what to use as separator. This means that you can consume any textual tabular format. Here is an example using ; as a separator:

type AirQuality = CsvProvider<"../data/AirQuality.csv", ";", ResolutionFolder=__SOURCE_DIRECTORY__>

let airQuality = new AirQuality()

for row in airQuality.Rows do
  if row.Month > 6 then
    printfn "Temp: %i Ozone: %f " row.Temp row.Ozone

The air quality dataset (data/AirQuality.csv) is used in many samples for the Statistical Computing language R. A short description of the dataset can be found in the R language manual.

If you are parsing a tab-separated file that uses \t as the separator, you can also specify the separator explicitly. However, if you're using an url or file that has the .tsv extension, the type provider will use \t by default. In the following example, we also set IgnoreErrors static parameter to true so that lines with incorrect number of elements are automatically skipped (the sample file (data/MortalityNY.csv) contains additional unstructured data at the end):

let mortalityNy = CsvProvider<"../data/MortalityNY.tsv", IgnoreErrors=true, ResolutionFolder=__SOURCE_DIRECTORY__>.GetSample()

// Find the name of a cause based on code
// (Pedal cyclist injured in an accident)
let cause = mortalityNy.Rows |> Seq.find (fun r ->
  r.``Cause of death Code`` = "V13.4")

// Print the number of injured cyclists
printfn "CAUSE: %s" cause.``Cause of death``
for r in mortalityNy.Rows do
  if r.``Cause of death Code`` = "V13.4" then
    printfn "%s (%d cases)" r.County r.Count

Finally, note that it is also possible to specify multiple different separators for the CsvProvider. This might be useful if a file is irregular and contains rows separated by either semicolon or a colon. You can use: CsvProvider<"../data/AirQuality.csv", Separator=";,", ResolutionFolder=__SOURCE_DIRECTORY__>.

Missing values

It is quite common in statistical datasets for some values to be missing. If you open the data/AirQuality.csv file you will see that some values for the ozone observations are marked #N/A. Such values are parsed as float and will be marked with Double.NaN in F#. The values NaN, NA, N/A, #N/A, :, -, TBA, and TBD are recognized as missing values by default, but you can customize it by specifying the MissingValues static parameter of CsvProvider as a comma-separated string. For example, to ignore this and that we could do:

seq [(nan, nan, 1.0M)]

The following snippet calculates the mean of the ozone observations excluding the Double.NaN values. We first obtain the Ozone property for each row, then remove missing values and then use the standard Seq.average function:

let mean =
  |> Seq.toArray
  |> Array.map (fun row -> row.Ozone)
  |> Array.filter (fun elem -> not (Double.IsNaN elem))
  |> Array.average

If the sample doesn't have missing values on all columns, but at runtime missing values could appear anywhere, you can set the static parameter AssumeMissingValues to true in order to force CsvProvider to assume missing values can occur in any column.

Controlling the column types

By default, the CSV type provider checks the first 1000 rows to infer the types, but you can customize it by specifying the InferRows static parameter of CsvProvider. If you specify 0 the entire file will be used.

Columns with only 0, 1, Yes, No, True, or False will be set to bool. Columns with numerical values will be set to either int, int64, decimal, or float, in that order of preference.

If a value is missing in any row, by default the CSV type provider will infer a nullable (for int and int64) or an optional (for bool, DateTime and Guid). When a decimal would be inferred but there are missing values, we will infer a float instead, and use Double.NaN to represent those missing values. The string type is already inherently nullable, so by default we won't generate a string option. If you prefer to use optionals in all cases, you can set the static parameter PreferOptionals to true. In that case you'll never get an empty string or a Double.NaN and will always get a None instead.

If you have other preferences, e.g. if you want a column to be a float instead of a decimal, you can override the default behaviour by specifying the types in the header column between braces, similar to what can be done to specify the units of measure. This will override both AssumeMissingValues and PreferOptionals. The valid types are:

You can also specify both the type and a unit (e.g float<metre>). Example:

Name,  Distance (decimal?<metre>), Time (float)
First, 50,                        3

Additionally, you can also specify some or all the types in the Schema static parameter of CsvProvider. Valid formats are:

What's specified in the Schema static parameter will always take precedence to what's specified in the column headers.

If the first row of the file is not a header row, you can specify the HasHeaders static parameter to false in order to consider that row as a data row. In that case, the columns will be named Column1, Column2, etc..., unless the names are overridden using the Schema parameter. Note that you can override only the name in the Schema parameter and still have the provider infer the type for you. Example:

type OneTwoThree =
  CsvProvider<"1,2,3", HasHeaders = false, Schema = "Duration (float<second>),foo,float option">

let csv = OneTwoThree.GetSample()
for row in csv.Rows do
  printfn "%f %d %f"
    (defaultArg row.Column3 1.0)

You don't need to override all the columns, you can skip the ones to leave as default. For example, in the titanic training dataset from Kaggle (data/Titanic.csv), if you want to rename the 3rd column (the PClass column) to Passenger Class and override the 6th column (the Fare column) to be a float instead of a decimal, you can define only that, and leave the other columns blank in the schema (you also don't need to add all the trailing commas).

type Titanic1 =
              Schema=",,Passenger Class,,,float",

let titanic1 = Titanic1.GetSample()
for row in titanic1.Rows do
  printfn "%s Class = %d Fare = %g"
    row.Name row.``Passenger Class`` row.Fare

Alternatively, you can rename and override the type of any column by name instead of by position:

type Titanic2 =
              Schema="Fare=float,PClass->Passenger Class",

let titanic2 = Titanic2.GetSample()
for row in titanic2.Rows do
  printfn "%s Class = %d Fare = %g"
    row.Name row.``Passenger Class`` row.Fare

You can even mix and match the two syntaxes like this Schema="int64,DidSurvive,PClass->Passenger Class=string"

Transforming CSV files

In addition to reading, CsvProvider also has support for transforming the row collection of CSV files. The operations available are Filter, Take, TakeWhile, Skip, SkipWhile, and Truncate. All these operations preserve the schema, so after transforming you can save the results by using one of the overloads of the Save method. You can also use the SaveToString() to get the output directly as a string.

// Saving the first 10 rows that don't have missing values to a new csv file
  .Filter(fun row ->
    not (Double.IsNaN row.Ozone) &&
    not (Double.IsNaN row.``Solar.R``))

It's also possible to transform the columns themselves by using Map and the constructor for the Row type.

let doubleOzone =
  airQuality.Map(fun row ->
      ( row.Ozone * 2.0, row.``Solar.R``,
        row.Wind, row.Temp, row.Month, row.Day))

You can also append new rows, either by creating them directly as in the previous example, or by parsing them from a string.

let newRows =

let airQualityWithExtraRows =
  airQuality.Append newRows

It's even possible to create csv files without parsing at all:

type MyCsvType =
  CsvProvider<Schema = "A (int), B (string), C (date option)",

let myRows =
  [ MyCsvType.Row(1, "a", None)
    MyCsvType.Row(2, "B", Some DateTime.Now) ]

let myCsv = new MyCsvType(myRows)

Handling big datasets

By default, the rows are cached so you can iterate over the Rows property multiple times without worrying. But if you will only iterate once, you can disable caching by setting the CacheRows static parameter of CsvProvider to false. If the number of rows is very big, you have to do this otherwise you may exhaust the memory. You can still cache the data at some point by using the Cache method, but only do that if you have already transformed the dataset to be smaller.

Related articles

Multiple items
namespace FSharp

namespace Microsoft.FSharp
Multiple items
namespace FSharp.Data

namespace Microsoft.FSharp.Data
type Stocks = CsvProvider<...>
type CsvProvider =
<summary>Typed representation of a CSV file.</summary> <param name='Sample'>Location of a CSV sample file or a string containing a sample CSV document.</param> <param name='Separators'>Column delimiter(s). Defaults to <c>,</c>.</param> <param name='InferRows'>Number of rows to use for inference. Defaults to <c>1000</c>. If this is zero, all rows are used.</param> <param name='Schema'>Optional column types, in a comma separated list. Valid types are <c>int</c>, <c>int64</c>, <c>bool</c>, <c>float</c>, <c>decimal</c>, <c>date</c>, <c>guid</c>, <c>string</c>, <c>int?</c>, <c>int64?</c>, <c>bool?</c>, <c>float?</c>, <c>decimal?</c>, <c>date?</c>, <c>guid?</c>, <c>int option</c>, <c>int64 option</c>, <c>bool option</c>, <c>float option</c>, <c>decimal option</c>, <c>date option</c>, <c>guid option</c> and <c>string option</c>. You can also specify a unit and the name of the column like this: <c>Name (type&lt;unit&gt;)</c>, or you can override only the name. If you don't want to specify all the columns, you can reference the columns by name like this: <c>ColumnName=type</c>.</param> <param name='HasHeaders'>Whether the sample contains the names of the columns as its first line.</param> <param name='IgnoreErrors'>Whether to ignore rows that have the wrong number of columns or which can't be parsed using the inferred or specified schema. Otherwise an exception is thrown when these rows are encountered.</param> <param name='SkipRows'>Skips the first n rows of the CSV file.</param> <param name='AssumeMissingValues'>When set to true, the type provider will assume all columns can have missing values, even if in the provided sample all values are present. Defaults to false.</param> <param name='PreferOptionals'>When set to true, inference will prefer to use the option type instead of nullable types, <c>double.NaN</c> or <c>""</c> for missing values. Defaults to false.</param> <param name='Quote'>The quotation mark (for surrounding values containing the delimiter). Defaults to <c>"</c>.</param> <param name='MissingValues'>The set of strings recogized as missing values specified as a comma-separated string (e.g., "NA,N/A"). Defaults to <c>NaN,NA,N/A,#N/A,:,-,TBA,TBD</c>.</param> <param name='CacheRows'>Whether the rows should be caches so they can be iterated multiple times. Defaults to true. Disable for large datasets.</param> <param name='Culture'>The culture used for parsing numbers and dates. Defaults to the invariant culture.</param> <param name='Encoding'>The encoding used to read the sample. You can specify either the character set name or the codepage number. Defaults to UTF8 for files, and to ISO-8859-1 the for HTTP requests, unless <c>charset</c> is specified in the <c>Content-Type</c> response header.</param> <param name='ResolutionFolder'>A directory that is used when resolving relative file references (at design time and in hosted execution).</param> <param name='EmbeddedResource'>When specified, the type provider first attempts to load the sample from the specified resource (e.g. 'MyCompany.MyAssembly, resource_name.csv'). This is useful when exposing types generated by the type provider.</param>
val msft : Runtime.CsvFile<CsvProvider<...>.Row>
CsvProvider<...>.Load(uri: string) : CsvProvider<...>
Loads CSV from the specified uri
CsvProvider<...>.Load(reader: System.IO.TextReader) : CsvProvider<...>
Loads CSV from the specified reader
CsvProvider<...>.Load(stream: System.IO.Stream) : CsvProvider<...>
Loads CSV from the specified stream
val firstRow : CsvProvider<...>.Row
property Runtime.CsvFile.Rows: seq<CsvProvider<...>.Row> with get
module Seq

from Microsoft.FSharp.Collections
val head : source:seq<'T> -> 'T
val lastDate : System.DateTime
property CsvProvider<...>.Row.Date: System.DateTime with get
val lastOpen : decimal
property CsvProvider<...>.Row.Open: decimal with get
val row : CsvProvider<...>.Row
val printfn : format:Printf.TextWriterFormat<'T> -> 'T
property CsvProvider<...>.Row.High: decimal with get
property CsvProvider<...>.Row.Low: decimal with get
property CsvProvider<...>.Row.Close: decimal with get
namespace XPlot
namespace XPlot.Plotly
namespace System
property CsvProvider<...>.Row.Date: DateTime with get
type Chart =
  static member Area : data:seq<#value> -> PlotlyChart + 2 overloads
  static member Bar : data:seq<#value> -> PlotlyChart + 2 overloads
  static member Bubble : data:seq<#key * #value * #value> -> PlotlyChart
  static member Candlestick : data:seq<#key * #value * #value * #value * #value> -> PlotlyChart
  static member Column : data:seq<#value> -> PlotlyChart + 2 overloads
  static member Line : data:seq<#value> -> PlotlyChart + 2 overloads
  static member Pie : data:seq<#key * #value> -> PlotlyChart
  static member Plot : data:Trace -> PlotlyChart + 3 overloads
  static member Scatter : data:seq<#value> -> PlotlyChart + 2 overloads
  static member Show : chart:PlotlyChart -> unit
static member Chart.Line : data:seq<#seq<'a1 * 'a2>> -> PlotlyChart (requires 'a1 :> key and 'a2 :> value)
static member Chart.Line : data:seq<#key * #value> -> PlotlyChart
static member Chart.Line : data:seq<#value> -> PlotlyChart
val recent : (DateTime * decimal * decimal * decimal * decimal) list
Multiple items
type DateTime =
  new : year: int * month: int * day: int -> unit + 10 overloads
  member Add : value: TimeSpan -> DateTime
  member AddDays : value: float -> DateTime
  member AddHours : value: float -> DateTime
  member AddMilliseconds : value: float -> DateTime
  member AddMinutes : value: float -> DateTime
  member AddMonths : months: int -> DateTime
  member AddSeconds : value: float -> DateTime
  member AddTicks : value: int64 -> DateTime
  member AddYears : value: int -> DateTime

DateTime ()
   (+0 other overloads)
DateTime(ticks: int64) : DateTime
   (+0 other overloads)
DateTime(ticks: int64, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, calendar: Globalization.Calendar) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, calendar: Globalization.Calendar) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, millisecond: int) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, millisecond: int, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime.Parse(s: string) : DateTime
DateTime.Parse(s: string, provider: IFormatProvider) : DateTime
DateTime.Parse(s: string, provider: IFormatProvider, styles: Globalization.DateTimeStyles) : DateTime
DateTime.Parse(s: ReadOnlySpan<char>,?provider: IFormatProvider,?styles: Globalization.DateTimeStyles) : DateTime
static member Chart.Candlestick : data:seq<#key * #value * #value * #value * #value> -> PlotlyChart
val small : CsvProvider<...>
val small2 : CsvProvider<...>
namespace Microsoft.FSharp.Data.UnitSystems
namespace Microsoft.FSharp.Data.UnitSystems.SI
namespace Microsoft.FSharp.Data.UnitSystems.SI.UnitNames
val speed : decimal<metre/UnitSystems.SI.UnitSymbols.s>
property CsvProvider<...>.Row.Distance: decimal<metre> with get
property CsvProvider<...>.Row.Time: decimal<UnitSystems.SI.UnitSymbols.s> with get
type metre
type second
property CsvProvider<...>.Row.Name: string with get
type AirQuality = CsvProvider<...>
val airQuality : AirQuality
property CsvProvider<...>.Row.Month: int with get
property CsvProvider<...>.Row.Temp: int with get
property CsvProvider<...>.Row.Ozone: float with get
val mortalityNy : CsvProvider<...>
val cause : CsvProvider<...>.Row
val find : predicate:('T -> bool) -> source:seq<'T> -> 'T
val r : CsvProvider<...>.Row
property CsvProvider<...>.Row.( Cause of death Code ): string with get
property CsvProvider<...>.Row.( Cause of death ): string with get
property CsvProvider<...>.Row.County: string with get
property CsvProvider<...>.Row.Count: int with get
val mean : float
val toArray : source:seq<'T> -> 'T []
type Array =
  interface ICollection
  interface IEnumerable
  interface IList
  interface IStructuralComparable
  interface IStructuralEquatable
  interface ICloneable
  new : unit -> unit
  member Clone : unit -> obj
  member CopyTo : array: Array * index: int -> unit + 1 overload
  member GetEnumerator : unit -> IEnumerator
val map : mapping:('T -> 'U) -> array:'T [] -> 'U []
val filter : predicate:('T -> bool) -> array:'T [] -> 'T []
val elem : float
val not : value:bool -> bool
type Double =
  member CompareTo : value: float -> int + 1 overload
  member Equals : obj: float -> bool + 1 overload
  member GetHashCode : unit -> int
  member GetTypeCode : unit -> TypeCode
  member System.IConvertible.ToBoolean : provider: IFormatProvider -> bool
  member System.IConvertible.ToByte : provider: IFormatProvider -> byte
  member System.IConvertible.ToChar : provider: IFormatProvider -> char
  member System.IConvertible.ToDateTime : provider: IFormatProvider -> DateTime
  member System.IConvertible.ToDecimal : provider: IFormatProvider -> decimal
  member System.IConvertible.ToDouble : provider: IFormatProvider -> float
Double.IsNaN(d: float) : bool
val average : array:'T [] -> 'T (requires member ( + ) and member DivideByInt and member get_Zero)
type OneTwoThree = CsvProvider<...>
val csv : CsvProvider<...>
CsvProvider<...>.GetSample() : CsvProvider<...>
property CsvProvider<...>.Row.Duration: float<second> with get
property CsvProvider<...>.Row.Foo: int with get
val defaultArg : arg:'T option -> defaultValue:'T -> 'T
property CsvProvider<...>.Row.Column3: Option<float> with get
type Titanic1 = CsvProvider<...>
val titanic1 : CsvProvider<...>
property CsvProvider<...>.Row.( Passenger Class ): int with get
property CsvProvider<...>.Row.Fare: decimal with get
type Titanic2 = CsvProvider<...>
val titanic2 : CsvProvider<...>
property CsvProvider<...>.Row.Fare: float with get
property CsvProvider<...>.Row.( Solar.R ): float with get
val doubleOzone : Runtime.CsvFile<CsvProvider<...>.Row>
member Runtime.CsvFile.Map : mapping:Func<'RowType,'RowType> -> Runtime.CsvFile<'RowType>
type Row =
  inherit float * float * decimal * int * int * int
  new : ozone: float * solarR: float * wind: decimal * temp: int * month: int * day: int -> Row
  member Day : int
  member Month : int
  member Ozone : float
  member ``Solar.R`` : float
  member Temp : int
  member Wind : decimal
property CsvProvider<...>.Row.Wind: decimal with get
property CsvProvider<...>.Row.Day: int with get
val newRows : CsvProvider<...>.Row []
CsvProvider<...>.ParseRows(text: string) : CsvProvider<...>.Row []
val airQualityWithExtraRows : Runtime.CsvFile<CsvProvider<...>.Row>
member Runtime.CsvFile.Append : rows:seq<'RowType> -> Runtime.CsvFile<'RowType>
type MyCsvType = CsvProvider<...>
val myRows : CsvProvider<...>.Row list
type Row =
  inherit int * string * Option<DateTime>
  new : a: int * b: string * c: Option<DateTime> -> Row
  member A : int
  member B : string
  member C : Option<DateTime>
union case Option.None: Option<'T>
union case Option.Some: Value: 'T -> Option<'T>
property DateTime.Now: DateTime with get
val myCsv : MyCsvType
member Runtime.CsvFile.SaveToString : ?separator:char * ?quote:char -> string