Usage and Examples

Before use

Attention

  1. The CSV parser requires all the lines have same number of fields/columns. Even lines with spaces will cause error.
  2. By default, csvtk thinks your files have header row, if not, switch flag -H on.
  3. Column names better be unique.
  4. By default, lines starting with # will be ignored, if the header row starts with #, please assign flag -C another rare symbol, e.g. '$'.
  5. By default, csvtk handles CSV files, use flag -t for tab-delimited files.
  6. If " exists in tab-delimited files, use flag -l.

Table of Contents

Information

Format conversion

Set operations

Edit

Ordering

Ploting

Misc

csvtk

Usage

csvtk -- a cross-platform, efficient and practical CSV/TSV toolkit

Version: 0.15.0

Author: Wei Shen <shenwei356@gmail.com>

Documents  : http://shenwei356.github.io/csvtk
Source code: https://github.com/shenwei356/csvtk

Attention:

    1. The CSV parser requires all the lines have same number of fields/columns.
       Even lines with spaces will cause error.
    2. By default, csvtk thinks your files have header row, if not, switch flag "-H" on.
    3. Column names better be unique.
    4. By default, lines starting with "#" will be ignored, if the header row
       starts with "#", please assign flag "-C" another rare symbol, e.g. '$'.
    5. By default, csvtk handles CSV files, use flag "-t" for tab-delimited files.
    6. If " exists in tab-delimited files, use flag "-l".

Environment variables for frequently used global flags

    - "CSVTK_T" for flag "-t/--tabs"
    - "CSVTK_H" for flag "-H/--no-header-row"

Usage:
  csvtk [command]

Available Commands:
  collapse        collapse one field with selected fields as keys
  concat          concatenate CSV/TSV files by rows
  csv2md          convert CSV to markdown format
  csv2tab         convert CSV to tabular format
  cut             select parts of fields
  filter          filter rows by values of selected fields with arithmetic expression
  filter2         filter rows by awk-like artithmetic/string expressions
  freq            frequencies of selected fields
  gather          gather columns into key-value pairs
  genautocomplete generate shell autocompletion script
  grep            grep data by selected fields with patterns/regular expressions
  head            print first N records
  headers         print headers
  help            Help about any command
  inter           intersection of multiple files
  join            join multiple CSV files by selected fields
  mutate          create new column from selected fields by regular expression
  mutate2         create new column from selected fields by awk-like artithmetic/string expressions
  plot            plot common figures
  pretty          convert CSV to readable aligned table
  rename          rename column names
  rename2         rename column names by regular expression
  replace         replace data of selected fields by regular expression
  sample          sampling by proportion
  sort            sort by selected fields
  space2tab       convert space delimited format to CSV
  split           split CSV/TSV into multiple files according to column values
  splitxlsx       split XLSX sheet into multiple sheets according to column values
  stats           summary of CSV file
  stats2          summary of selected digital fields
  tab2csv         convert tabular format to CSV
  transpose       transpose CSV data
  uniq            unique data without sorting
  version         print version information and check for update
  xlsx2csv        convert XLSX to CSV format

Flags:
  -c, --chunk-size int         chunk size of CSV reader (default 50)
  -C, --comment-char string    lines starting with commment-character will be ignored. if your header row starts with '#', please assign "-C" another rare symbol, e.g. '$' (default "#")
  -d, --delimiter string       delimiting character of the input CSV file (default ",")
  -h, --help                   help for csvtk
  -E, --ignore-empty-row       ignore empty row
  -l, --lazy-quotes            if given, a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field
  -H, --no-header-row          specifies that the input CSV file does not have header row
  -j, --num-cpus int           number of CPUs to use (default value depends on your computer) (default 4)
  -D, --out-delimiter string   delimiting character of the output CSV file (default ",")
  -o, --out-file string        out file ("-" for stdout, suffix .gz for gzipped out) (default "-")
  -T, --out-tabs               specifies that the output is delimited with tabs. Overrides "-D"
  -t, --tabs                   specifies that the input CSV file is delimited with tabs. Overrides "-d" and "-D"

headers

Usage

print headers

Usage:
  csvtk headers [flags]

Examples

$ csvtk headers testdata/*.csv$
# testdata/1.csv
1       name
2       attr
# testdata/2.csv
1       name
2       major
# testdata/3.csv
1       id
2       name
3       hobby

stats

Usage

summary of CSV file

Usage:
  csvtk stats [flags]

Aliases:
  stats, stat

Examples

  1. with header row

    $ cat testdata/names.csv
    id,first_name,last_name,username
    11,"Rob","Pike",rob
    2,Ken,Thompson,ken
    4,"Robert","Griesemer","gri"
    1,"Robert","Thompson","abc"
    NA,"Robert","Abel","123"
    
    $ cat testdata/names.csv | csvtk stats
    file   num_cols   num_rows
    -             4          5
    
  2. no header row

    $ cat  testdata/digitals.tsv
    4       5       6
    1       2       3
    7       8       0
    8       1,000   4
    
    $ cat  testdata/digitals.tsv | csvtk stats -t -H
    file   num_cols   num_rows
    -             3          4
    

stat2

Usage

summary of selected digital fields: num, sum, min, max, mean, stdev

Usage:
  csvtk stats2 [flags]

Flags:
  -f, --fields string   select only these fields. e.g -f 1,2 or -f columnA,columnB
  -F, --fuzzy-fields    using fuzzy fields, e.g., -F -f "*name" or -F -f "id123*"

Examples

  1. simplest one

    $ seq 1 5 | csvtk stats2 -H -f 1
    field   num   sum   min   max   mean   stdev
    1         5    15     1     5      3    1.58
    
  2. multiple fields

    $ cat  testdata/digitals.tsv
    4       5       6
    1       2       3
    7       8       0
    8       1,000   4
    
    $ cat  testdata/digitals.tsv | csvtk stats2 -t -H -f 1-3
    field   num     sum   min     max     mean    stdev
    1         4      20     1       8        5     3.16
    2         4   1,015     2   1,000   253.75   497.51
    3         4      13     0       6     3.25      2.5
    

pretty

Usage

convert CSV to readable aligned table

Attention:

    csv2md treats the first row as header line and requires them to be unique

Usage:
  csvtk pretty [flags]

Flags:
  -r, --align-right        align right
  -W, --max-width int      max width
  -w, --min-width int      min width
  -s, --separator string   fields/columns separator (default "   ")

Examples:

  1. default

    $ csvtk pretty testdata/names.csv
    id   first_name   last_name   username
    11   Rob          Pike        rob
    2    Ken          Thompson    ken
    4    Robert       Griesemer   gri
    1    Robert       Thompson    abc
    NA   Robert       Abel        123
    
  2. align right

    $ csvtk pretty testdata/names.csv -r
    id   first_name   last_name   username
    11          Rob        Pike        rob
     2          Ken    Thompson        ken
     4       Robert   Griesemer        gri
     1       Robert    Thompson        abc
    NA       Robert        Abel        123
    
  3. custom separator

    $ csvtk pretty testdata/names.csv -s " | "
    id | first_name | last_name | username
    11 | Rob        | Pike      | rob
    2  | Ken        | Thompson  | ken
    4  | Robert     | Griesemer | gri
    1  | Robert     | Thompson  | abc
    NA | Robert     | Abel      | 123
    

transpose

Usage

transpose CSV data

Usage:
  csvtk transpose [flags]

Examples

$ cat  testdata/digitals.tsv
4       5       6$
1       2       3$
7       8       0
8       1,000   4

$ csvtk transpose -t  testdata/digitals.tsv
4       1       7       8
5       2       8       1,000
6       3       0       4

csv2md

Usage

convert CSV to markdown format

Attention:

    csv2md treats the first row as header line and requires them to be unique

Usage:
  csvtk csv2md [flags]

Flags:
  -a, --alignments string   comma separated alignments. e.g. -a l,c,c,c or -a c (default "l")
  -w, --min-width int       min width (at least 3) (default 3)

Examples

  1. give single alignment symbol

    $ cat testdata/names.csv | csvtk csv2md -a left
    id |first_name|last_name|username
    :--|:---------|:--------|:-------
    11 |Rob       |Pike     |rob
    2  |Ken       |Thompson |ken
    4  |Robert    |Griesemer|gri
    1  |Robert    |Thompson |abc
    NA |Robert    |Abel     |12
    

    result:

    id first_name last_name username
    11 Rob Pike rob
    2 Ken Thompson ken
    4 Robert Griesemer gri
    1 Robert Thompson abc
    NA Robert Abel 12
  2. give alignment symbols of all fields

    $ cat testdata/names.csv | csvtk csv2md -a c,l,l,l
    id |first_name|last_name|username
    :-:|:---------|:--------|:-------
    11 |Rob       |Pike     |rob
    2  |Ken       |Thompson |ken
    4  |Robert    |Griesemer|gri
    1  |Robert    |Thompson |abc
    NA |Robert    |Abel     |123
    

    result

    id first_name last_name username
    11 Rob Pike rob
    2 Ken Thompson ken
    4 Robert Griesemer gri
    1 Robert Thompson abc
    NA Robert Abel 123

xlsx2csv

Usage

convert XLSX to CSV format

Usage:
  csvtk xlsx2csv [flags]

Flags:
  -h, --help                help for xlsx2csv
  -a, --list-sheets         list all sheets
  -i, --sheet-index int     Nth sheet to retrieve (default 1)
  -n, --sheet-name string   sheet to retrieve

Examples

  1. list all sheets

    $ csvtk xlsx2csv ../testdata/accounts.xlsx -a
    index   sheet
    1       names
    2       phones
    3       region
    
  2. retrieve sheet by index

    $ csvtk xlsx2csv ../testdata/accounts.xlsx -i 3
    name,region
    ken,nowhere
    gri,somewhere
    shenwei,another
    Thompson,there
    
  3. retrieve sheet by name

    $ csvtk xlsx2sv ../testdata/accounts.xlsx -n region
    name,region
    ken,nowhere
    gri,somewhere
    shenwei,another
    Thompson,there
    

Usage

print first N records

Usage:
  csvtk head [flags]

Flags:
  -n, --number int   print first N records (default 10)

Examples

  1. with header line

    $ csvtk head -n 2 testdata/1.csv
    name,attr
    foo,cool
    bar,handsome
    
  2. no header line

    $ csvtk head -H -n 2 testdata/1.csv
    name,attr
    foo,cool
    

concat

Usage

concatenate CSV/TSV files by rows

Note that the second and later files are concatenated to the first one,
so only columns match that of the first files kept.

Usage:
  csvtk concat [flags]

Flags:
  -h, --help                    help for concat
  -i, --ignore-case             ignore case (column name)
  -k, --keep-unmatched          keep blanks even if no any data of a file matches
  -u, --unmatched-repl string   replacement for unmatched data

Examples

  1. data

    $ csvtk pretty names.csv
    id   first_name   last_name   username
    11   Rob          Pike        rob
    2    Ken          Thompson    ken
    4    Robert       Griesemer   gri
    1    Robert       Thompson    abc
    NA   Robert       Abel        123
    
    $ csvtk pretty  names.reorder.csv
    last_name   username   id   first_name
    Pike        rob        11   Rob
    Thompson    ken        2    Ken
    Griesemer   gri        4    Robert
    Thompson    abc        1    Robert
    Abel        123        NA   Robert
    
    $ csvtk pretty  names.with-unmatched-colname.csv
    id2   First_name   Last_name    Username   col
    22    Rob33        Pike222      rob111     abc
    44    Ken33        Thompson22   ken111     def
    
  2. simple one

    $ csvtk concat names.csv names.reorder.csv | csvtk pretty
    id   first_name   last_name   username
    11   Rob          Pike        rob
    2    Ken          Thompson    ken
    4    Robert       Griesemer   gri
    1    Robert       Thompson    abc
    NA   Robert       Abel        123
    11   Rob          Pike        rob
    2    Ken          Thompson    ken
    4    Robert       Griesemer   gri
    1    Robert       Thompson    abc
    NA   Robert       Abel        123
    
  3. data with unmatched column names, and ignoring cases

    $ csvtk concat names.csv names.with-unmatched-colname.csv -i | csvtk pretty
    id   first_name   last_name    username
    11   Rob          Pike         rob
    2    Ken          Thompson     ken
    4    Robert       Griesemer    gri
    1    Robert       Thompson     abc
    NA   Robert       Abel         123
         Rob33        Pike222      rob111
         Ken33        Thompson22   ken111
    
     $ csvtk concat names.csv names.with-unmatched-colname.csv -i -u Unmached | csvtk pretty
     id         first_name   last_name    username
     11         Rob          Pike         rob
     2          Ken          Thompson     ken
     4          Robert       Griesemer    gri
     1          Robert       Thompson     abc
     NA         Robert       Abel         123
     Unmached   Rob33        Pike222      rob111
     Unmached   Ken33        Thompson22   ken111
    
  4. Sometimes data of one file does not matche any column, they are discared by default. But you can keep them using flag -k/--keep-unmatched

    $ csvtk concat names.with-unmatched-colname.csv names.csv | csvtk pretty
    id2   First_name   Last_name    Username   col
    22    Rob33        Pike222      rob111     abc
    44    Ken33        Thompson22   ken111     def
    
    $ csvtk concat names.with-unmatched-colname.csv names.csv -u -k NA | csvtk pretty
    id2   First_name   Last_name    Username   col
    22    Rob33        Pike222      rob111     abc
    44    Ken33        Thompson22   ken111     def
    NA    NA           NA           NA         NA
    NA    NA           NA           NA         NA
    NA    NA           NA           NA         NA
    NA    NA           NA           NA         NA
    NA    NA           NA           NA         NA
    

sample

Usage

sampling by proportion

Usage:
  csvtk sample [flags]

Flags:
  -p, --proportion float   sample by proportion
  -s, --rand-seed int      rand seed (default 11)

Examples

$ seq 100 | csvtk sample -H -p 0.5 | wc -l
46

$ seq 100 | csvtk sample -H -p 0.5 | wc -l
46

$ seq 100 | csvtk sample -H -p 0.1 | wc -l
10

$ seq 100 | csvtk sample -H -p 0.1 | wc -l
10

cut

Usage

select parts of fields

Usage:
  csvtk cut [flags]

Flags:
  -f, --fields string   select only these fields. e.g -f 1,2 or -f columnA,columnB, or -f -columnA for unselect columnA
  -F, --fuzzy-fields    using fuzzy fields, e.g., -F -f "*name" or -F -f "id123*"
  -h, --help            help for cut
  -i, --ignore-case     ignore case (column name)

Examples

uniq

Usage

unique data without sorting

Usage:
  csvtk uniq [flags]

Flags:
  -f, --fields string   select only these fields. e.g -f 1,2 or -f columnA,columnB (default "1")
  -F, --fuzzy-fields    using fuzzy fields, e.g., -F -f "*name" or -F -f "id123*"
  -i, --ignore-case     ignore case

Examples:

freq

Usage

frequencies of selected fields

Usage:
  csvtk freq [flags]

Flags:
  -f, --fields string   select only these fields. e.g -f 1,2 or -f columnA,columnB (default "1")
  -F, --fuzzy-fields    using fuzzy fields, e.g., -F -f "*name" or -F -f "id123*"
  -i, --ignore-case     ignore case
  -r, --reverse         reverse order while sorting
  -n, --sort-by-freq    sort by frequency
  -k, --sort-by-key     sort by key

Examples

  1. one filed

    $ cat testdata/names.csv | csvtk freq -f first_name | csvtk pretty
    first_name   frequency
    Ken          1
    Rob          1
    Robert       3
    
  2. sort by frequency. you can also use csvtk sort with more sorting options

    $ cat testdata/names.csv | csvtk freq -f first_name -n -r | csvtk pretty
    first_name   frequency
    Robert       3
    Ken          1
    Rob          1
    
  3. sorty by key

    $ cat testdata/names.csv | csvtk freq -f first_name -k | csvtk pretty
    first_name   frequency
    Ken          1
    Rob          1
    Robert       3
    
  4. multiple fields

    $ cat testdata/names.csv | csvtk freq -f first_name,last_name | csvtk pretty
    first_name   last_name   frequency
    Robert       Abel        1
    Ken          Thompson    1
    Rob          Pike        1
    Robert       Thompson    1
    Robert       Griesemer   1
    
  5. data without header row

    $ cat testdata/ testdata/digitals.tsv  | csvtk -t -H freq -f 1
    8       1
    1       1
    4       1
    7       1
    

inter

Usage

intersection of multiple files

Usage:
  csvtk inter [flags]

Flags:
  -f, --fields string   select only these fields. e.g -f 1,2 or -f columnA,columnB (default "1")
  -F, --fuzzy-fields    using fuzzy fields, e.g., -F -f "*name" or -F -f "id123*"
  -i, --ignore-case     ignore case

Examples:

$ cat testdata/phones.csv
username,phone
gri,11111
rob,12345
ken,22222
shenwei,999999

$ cat testdata/region.csv
name,region
ken,nowhere
gri,somewhere
shenwei,another
Thompson,there

$ csvtk inter testdata/phones.csv testdata/region.csv
username
gri
ken
shenwei

grep

Usage

grep data by selected fields with patterns/regular expressions

Usage:
  csvtk grep [flags]

Flags:
  -f, --fields string         comma separated key fields, column name or index. e.g. -f 1-3 or -f id,id2 or -F -f "group*" (default "1")
  -F, --fuzzy-fields          using fuzzy fields, e.g., -F -f "*name" or -F -f "id123*"
  -h, --help                  help for grep
  -i, --ignore-case           ignore case
  -v, --invert                invert match
  -n, --line-number           print line number as the first column ("n")
  -N, --no-highlight          no highlight
  -p, --pattern strings       query pattern (multiple values supported)
  -P, --pattern-file string   pattern files (one pattern per line)
  -r, --use-regexp            patterns are regular expression
      --verbose               verbose output

Examples

Matched parts will be highlight

filter

Usage

filter rows by values of selected fields with arithmetic expression

Usage:
  csvtk filter [flags]

Flags:
      --any             print record if any of the field satisfy the condition
  -f, --filter string   filter condition. e.g. -f "age>12" or -f "1,3<=2" or -F -f "c*!=0"
  -F, --fuzzy-fields    using fuzzy fields, e.g., -F -f "*name" or -F -f "id123*"
  -h, --help            help for filter
  -n, --line-number     print line number as the first column ("n")

Examples

  1. single field

    $ cat testdata/names.csv
    id,first_name,last_name,username
    11,"Rob","Pike",rob
    2,Ken,Thompson,ken
    4,"Robert","Griesemer","gri"
    1,"Robert","Thompson","abc"
    NA,"Robert","Abel","123"
    
    $ cat testdata/names.csv | csvtk filter -f "id>0" | csvtk pretty
    id   first_name   last_name   username
    11   Rob          Pike        rob
    2    Ken          Thompson    ken
    4    Robert       Griesemer   gri
    1    Robert       Thompson    abc
    
  2. multiple fields

    $ cat  testdata/digitals.tsv
    4       5       6
    1       2       3
    7       8       0
    8       1,000   4
    
    $ cat  testdata/digitals.tsv | csvtk -t -H filter -f "1-3>0"
    4       5       6
    1       2       3
    8       1,000   4
    

    using --any to print record if any of the field satisfy the condition

    $  cat  testdata/digitals.tsv | csvtk -t -H filter -f "1-3>0" --any
    4       5       6
    1       2       3
    7       8       0
    8       1,000   4
    
  3. fuzzy fields

    $  cat testdata/names.csv | csvtk filter -F -f "i*!=0"
    id,first_name,last_name,username
    11,Rob,Pike,rob
    2,Ken,Thompson,ken
    4,Robert,Griesemer,gri
    1,Robert,Thompson,abc
    

filter2

Usage

filter rows by awk-like arithmetic/string expressions

The arithmetic/string expression is supported by:

    https://github.com/Knetic/govaluate

Supported operators and types:

    Modifiers: + - / * & | ^ ** % >> <<
    Comparators: > >= < <= == != =~ !~
    Logical ops: || &&
    Numeric constants, as 64-bit floating point (12345.678)
    String constants (single quotes: 'foobar')
    Date constants (single quotes)
    Boolean constants: true false
    Parenthesis to control order of evaluation ( )
    Arrays (anything separated by , within parenthesis: (1, 2, 'foo'))
    Prefixes: ! - ~
    Ternary conditional: ? :
    Null coalescence: ??

Usage:
  csvtk filter2 [flags]

Flags:
  -f, --filter string   awk-like filter condition. e.g. '$age>12' or '$1 > $3' or '$name=="abc"' or '$1 % 2 == 0'
  -h, --help            help for filter2
  -n, --line-number     print line number as the first column ("n")


Examples:

  1. filter rows with id greater than 3:

    $ cat testdata/names.csv
    id,first_name,last_name,username
    11,"Rob","Pike",rob
    2,Ken,Thompson,ken
    4,"Robert","Griesemer","gri"
    1,"Robert","Thompson","abc"
    NA,"Robert","Abel","123"
    
    $ cat testdata/names.csv | csvtk filter2 -f '$id > 3'
    id,first_name,last_name,username
    11,Rob,Pike,rob
    4,Robert,Griesemer,gri
    
  2. arithmetic and string expressions

    $ cat testdata/names.csv | csvtk filter2 -f '$id > 3 || $username=="ken"'
    id,first_name,last_name,username
    11,Rob,Pike,rob
    2,Ken,Thompson,ken
    4,Robert,Griesemer,gri
    
  3. More arithmetic expressions

    $ cat testdata/digitals.tsv
    4       5       6
    1       2       3
    7       8       0
    8       1,000   4
    
    $ cat testdata/digitals.tsv | csvtk filter2 -H -t -f '$1 > 2 && $2 % 2 == 0'
    7       8       0
    8       1,000   4
    
    # comparison between fields and support
    $ cat testdata/digitals.tsv | csvtk filter2 -H -t -f '$2 <= $3 || ( $1 / $2 > 0.5 )'
    4       5       6
    1       2       3
    7       8       0
    

join

Usage

join 2nd and later files to the first file by selected fields.

Multiple keys supported, but the orders are ignored.

Usage:
  csvtk join [flags]

Flags:
  -f, --fields string    Semicolon separated key fields of all files, if given one, we think all the files have the same key columns. Fields of different files should be separated by ";", e.g -f "1;2" or -f "A,B;C,D" or -f id (default "1")
      --fill string      fill content for unmatched data
  -F, --fuzzy-fields     using fuzzy fields, e.g., -F -f "*name" or -F -f "id123*"
  -i, --ignore-case      ignore case
  -k, --keep-unmatched   keep unmatched data of the first file

Examples:

split

Usage

split CSV/TSV into multiple files according to column values

Note:

  1. flag -o/--out-file can specify out directory for splitted files

Usage:
  csvtk split [flags]

Flags:
  -g, --buf-groups int   buffering N groups before writing to file (default 100)
  -b, --buf-rows int     buffering N rows for every group before writing to file (default 100000)
  -f, --fields string    comma separated key fields, column name or index. e.g. -f 1-3 or -f id,id2 or -F -f "group*" (default "1")
  -F, --fuzzy-fields     using fuzzy fields, e.g., -F -f "*name" or -F -f "id123*"
  -h, --help             help for split
  -i, --ignore-case      ignore case
  -G, --out-gzip         force output gzipped file

Examples

  1. Test data

    $ cat names.csv
    id,first_name,last_name,username
    11,"Rob","Pike",rob
    2,Ken,Thompson,ken
    4,"Robert","Griesemer","gri"
    1,"Robert","Thompson","abc"
    NA,"Robert","Abel","123"
    
  2. split according to first_name

    $ csvtk split names.csv -f first_name
    $ ls *.csv
    names.csv  names-Ken.csv  names-Rob.csv  names-Robert.csv
    
    $ cat names-Ken.csv
    id,first_name,last_name,username
    2,Ken,Thompson,ken
    
    $ cat names-Rob.csv
    id,first_name,last_name,username
    11,Rob,Pike,rob
    
    $ cat names-Robert.csv
    id,first_name,last_name,username
    4,Robert,Griesemer,gri
    1,Robert,Thompson,abc
    NA,Robert,Abel,123
    
  3. split according to first_name and last_name

    $ csvtk split names.csv -f first_name,last_name
    $ ls *.csv
    names.csv               names-Robert-Abel.csv       names-Robert-Thompson.csv
    names-Ken-Thompson.csv  names-Robert-Griesemer.csv  names-Rob-Pike.csv
    
  4. flag `-o/--out-file can specify out directory for splitted files

    $ seq 10000 | csvtk split -H -o result
    $ ls result/*.csv | wc -l
    10000
    
  5. extreme example 1: lots (1M) of rows in groups

    $ yes 2 | head -n 10000000 | gzip -c > t.gz
    
    $ memusg -t ./csvtk -H split t.gz
    elapsed time: 7.959s
    peak rss: 35.7 MB
    
    # check
    $ zcat t-2.gz | wc -l
    10000000
    $ zcat t-2.gz | md5sum
    f194afd7cecf645c0e3cce50c9bc526e  -
    $ zcat t.gz | md5sum
    f194afd7cecf645c0e3cce50c9bc526e  -
    
  6. extreme example 2: lots (10K) of groups

    $ seq 10000 | gzip -c > t2.gz
    
    $ memusg -t ./csvtk -H split t2.gz  -o t2
    elapsed time: 20.856s
    peak rss: 23.77 MB
    
    # check
    $ ls t2/*.gz | wc -l
    10000
    $ zcat t2/*.gz | sort -k 1,1n | md5sum
    72d4ff27a28afbc066d5804999d5a504  -
    $ zcat t2.gz | md5sum
    72d4ff27a28afbc066d5804999d5a504  -
    

splitxlsx

Usage

split XLSX sheet into multiple sheets according to column values

Strengths: Sheet properties are remained unchanged.
Weakness : Complicated sheet structures are not well supported, e.g.,
1. merged cells
2. more than one header row

Usage:
  csvtk splitxlsx [flags]

Flags:
  -f, --fields string       comma separated key fields, column name or index. e.g. -f 1-3 or -f id,id2 or -F -f "group*" (default "1")
  -F, --fuzzy-fields        using fuzzy fields, e.g., -F -f "*name" or -F -f "id123*"
  -h, --help                help for splitxlsx
  -i, --ignore-case         ignore case (cell value)
  -a, --list-sheets         list all sheets
  -I, --sheet-index int     Nth sheet to retrieve (default 1)
  -n, --sheet-name string   sheet to retrieve

Examples

  1. example data

    # list all sheets
    $ csvtk xlsx2csv -a accounts.xlsx
    index   sheet
    1       names
    2       phones
    3       region
    
    # data of sheet "names"
    $ csvtk xlsx2csv accounts.xlsx | csvtk pretty
    id   first_name   last_name   username
    11   Rob          Pike        rob
    2    Ken          Thompson    ken
    4    Robert       Griesemer   gri
    1    Robert       Thompson    abc
    NA   Robert       Abel        123
    
  2. split sheet "names" according to first_name

    $ csvtk splitxlsx accounts.xlsx -n names -f first_name
    
    $ ls accounts.*
    accounts.split.xlsx  accounts.xlsx
    
    $ csvtk splitxlsx -a accounts.split.xlsx
    index   sheet
    1       names
    2       phones
    3       region
    4       Rob
    5       Ken
    6       Robert
    
    $ csvtk xlsx2csv accounts.split.xlsx -n Rob | csvtk pretty
    id   first_name   last_name   username
    11   Rob          Pike        rob
    
    $ csvtk xlsx2csv accounts.split.xlsx -n Robert | csvtk pretty
    id   first_name   last_name   username
    4    Robert       Griesemer   gri
    1    Robert       Thompson    abc
    NA   Robert       Abel        123
    

collapse

Usage

collapse one field with selected fields as keys

Usage:
  csvtk collapse [flags]

Flags:
  -f, --fields string      key fields. e.g -f 1,2 or -f columnA,columnB (default "1")
  -F, --fuzzy-fields       using fuzzy fields (only for key fields), e.g., -F -f "*name" or -F -f "id123*"
  -h, --help               help for collapse
  -i, --ignore-case        ignore case
  -s, --separater string   separater for collapsed data (default ";")
  -v, --vfield string      value field

examples

  1. data

    $ csvtk pretty teachers.csv
    lab                     teacher   class
    computational biology   Tom       Bioinformatics
    computational biology   Tom       Statistics
    computational biology   Rob       Bioinformatics
    sequencing center       Jerry     Bioinformatics
    sequencing center       Nick      Molecular Biology
    sequencing center       Nick      Microbiology
    
  2. List teachers for every lab/class. uniq is used to deduplicate items.

    $ cat teachers.csv  \
        | csvtk uniq -f lab,teacher  \
        | csvtk collapse -f lab -v teacher \
        | csvtk pretty
    
    lab                     teacher
    computational biology   Tom;Rob
    sequencing center       Jerry;Nick
    
    $ cat teachers.csv  \
        | csvtk uniq -f class,teacher  \
        | csvtk collapse -f class -v teacher -s ", " \
        | csvtk pretty
    
    class               teacher
    Statistics          Tom
    Bioinformatics      Tom, Rob, Jerry
    Molecular Biology   Nick
    Microbiology        Nick
    
  3. Multiple key fields supported

    $ cat teachers.csv  \
        | csvtk collapse -f teacher,lab -v class \
        | csvtk pretty
    
    teacher   lab                     class
    Tom       computational biology   Bioinformatics;Statistics
    Rob       computational biology   Bioinformatics
    Jerry     sequencing center       Bioinformatics
    Nick      sequencing center       Molecular Biology;Microbiology
    

rename

Usage

rename column names

Usage:
  csvtk rename [flags]

Flags:
  -f, --fields string   select only these fields. e.g -f 1,2 or -f columnA,columnB
  -F, --fuzzy-fields    using fuzzy fields, e.g., -F -f "*name" or -F -f "id123*"
  -n, --names string    comma separated new names

Examples:

rename2

Usage

rename column names by regular expression

Special replacement symbols:

    {nr}  ascending number, starting from 1
    {kv}  Corresponding value of the key (captured variable $n) by key-value file,
          n can be specified by flag -I (--key-capt-idx) (default: 1)

Usage:
  csvtk rename2 [flags]

Flags:
  -f, --fields string          select only these fields. e.g -f 1,2 or -f columnA,columnB
  -F, --fuzzy-fields           using fuzzy fields, e.g., -F -f "*name" or -F -f "id123*"
  -h, --help                   help for rename2
  -i, --ignore-case            ignore case
  -K, --keep-key               keep the key as value when no value found for the key
  -I, --key-capt-idx int       capture variable index of key (1-based) (default 1)
      --key-miss-repl string   replacement for key with no corresponding value
  -k, --kv-file string         tab-delimited key-value file for replacing key with value when using "{kv}" in -r (--replacement)
  -p, --pattern string         search regular expression
  -r, --replacement string     renamement. supporting capture variables.  e.g. $1 represents the text of the first submatch. ATTENTION: use SINGLE quote NOT double quotes in *nix OS or use the \ escape character. Ascending number is also supported by "{nr}".use ${1} instead of $1 when {kv} given!

Examples:

replace

Usage

replace data of selected fields by regular expression

Note that the replacement supports capture variables.
e.g. $1 represents the text of the first submatch.
ATTENTION: use SINGLE quote NOT double quotes in *nix OS.

Examples: Adding space to all bases.

    csvtk replace -p "(.)" -r '$1 ' -s

Or use the \ escape character.

    csvtk replace -p "(.)" -r "\$1 " -s

more on: http://shenwei356.github.io/csvtk/usage/#replace

Special replacement symbols:

        {nr}    Record number, starting from 1
        {kv}    Corresponding value of the key (captured variable $n) by key-value file,
                n can be specified by flag -I (--key-capt-idx) (default: 1)

Usage:
  csvtk replace [flags]

Flags:
  -f, --fields string          select only these fields. e.g -f 1,2 or -f columnA,columnB (default "1")
  -F, --fuzzy-fields           using fuzzy fields, e.g., -F -f "*name" or -F -f "id123*"
  -i, --ignore-case            ignore case
  -K, --keep-key               keep the key as value when no value found for the key
  -I, --key-capt-idx int       capture variable index of key (1-based) (default 1)
      --key-miss-repl string   replacement for key with no corresponding value
  -k, --kv-file string         tab-delimited key-value file for replacing key with value when using "{kv}" in -r (--replacement)
  -p, --pattern string         search regular expression
  -r, --replacement string     replacement. supporting capture variables.  e.g. $1 represents the text of the first submatch. ATTENTION: for *nix OS, use SINGLE quote NOT double quotes or use the \ escape character. Record number is also supported by "{nr}".use ${1} instead of $1 when {kv} given!

Examples

mutate

Usage

create new column from selected fields by regular expression

Usage:
  csvtk mutate [flags]

Flags:
  -f, --fields string    select only these fields. e.g -f 1,2 or -f columnA,columnB (default "1")
  -i, --ignore-case      ignore case
      --na               for unmatched data, use blank instead of original data
  -n, --name string      new column name
  -p, --pattern string   search regular expression with capture bracket. e.g. (default "^(.+)$")

Examples

mutate2

Usage

create new column from selected fields by awk-like arithmetic/string expressions

The arithmetic/string expression is supported by:

    https://github.com/Knetic/govaluate

Supported operators and types:

    Modifiers: + - / * & | ^ ** % >> <<
    Comparators: > >= < <= == != =~ !~
    Logical ops: || &&
    Numeric constants, as 64-bit floating point (12345.678)
    String constants (single quotes: 'foobar')
    Date constants (single quotes)
    Boolean constants: true false
    Parenthesis to control order of evaluation ( )
    Arrays (anything separated by , within parenthesis: (1, 2, 'foo'))
    Prefixes: ! - ~
    Ternary conditional: ? :
    Null coalescence: ??

Usage:
  csvtk mutate2 [flags]

Flags:
  -L, --digits int          number of digits after the dot (default 2)
  -s, --digits-as-string    treate digits as string to avoid converting big digits into scientific notation
  -e, --expression string   arithmetic/string expressions. e.g. "'string'", '"abc"', ' $a + "-" + $b ', '$1 + $2', '$a / $b', ' $1 > 100 ? "big" : "small" '
  -h, --help                help for mutate2
  -n, --name string         new column name

Example

  1. Constants

    $ cat testdata/digitals.tsv | csvtk mutate2 -t -H -e " 'abc' "
    4       5       6       abc
    1       2       3       abc
    7       8       0       abc
    8       1,000   4       abc
    
    $ val=123 && cat testdata/digitals.tsv | csvtk mutate2 -t -H -e " $val "
    4       5       6       123
    1       2       3       123
    7       8       0       123
    8       1,000   4       123
    
  2. String concatenation

    $ cat testdata/names.csv  \
        | csvtk mutate2 -n full_name -e ' $first_name + " " + $last_name ' \
        | csvtk pretty
    id   first_name   last_name   username   full_name
    11   Rob          Pike        rob        Rob Pike
    2    Ken          Thompson    ken        Ken Thompson
    4    Robert       Griesemer   gri        Robert Griesemer
    1    Robert       Thompson    abc        Robert Thompson
    NA   Robert       Abel        123        Robert Abel
    
  3. Math

    $ cat testdata/digitals.tsv | csvtk mutate2 -t -H -e '$1 + $3' -L 0
    4       5       6       10
    1       2       3       4
    7       8       0       7
    8       1,000   4       12
    
  4. Bool

    $ cat testdata/digitals.tsv | csvtk mutate2 -t -H -e '$1 > 5'
    4       5       6       false
    1       2       3       false
    7       8       0       true
    8       1,000   4       true
    
  5. Ternary conditional

    $ cat testdata/digitals.tsv | csvtk mutate2 -t -H -e '$1 > 5 ? "big" : "small" '
    4       5       6       small
    1       2       3       small
    7       8       0       big
    8       1,000   4       big
    

gather

Usage

gather columns into key-value pairs

Usage:
  csvtk gather [flags]

Flags:
  -f, --fields string   fields for gathering. e.g -f 1,2 or -f columnA,columnB, or -f -columnA for unselect columnA
  -F, --fuzzy-fields    using fuzzy fields, e.g., -F -f "*name" or -F -f "id123*"
  -k, --key string      name of key column to create in output
  -v, --value string    name of value column to create in output

Examples:

$ cat testdata/names.csv
id,first_name,last_name,username
11,"Rob","Pike",rob
2,Ken,Thompson,ken
4,"Robert","Griesemer","gri"
1,"Robert","Thompson","abc"
NA,"Robert","Abel","123

$ cat testdata/names.csv | csvtk gather -k item -v value -f -1
id,item,value
11,first_name,Rob
11,last_name,Pike
11,username,rob
2,first_name,Ken
2,last_name,Thompson
2,username,ken
4,first_name,Robert
4,last_name,Griesemer
4,username,gri
1,first_name,Robert
1,last_name,Thompson
1,username,abc
NA,first_name,Robert
NA,last_name,Abel
NA,username,123

sort

Usage

sort by selected fields

Usage:
  csvtk sort [flags]

Flags:
  -h, --help             help for sort
  -i, --ignore-case      ignore-case
  -k, --keys strings     keys (multiple values supported). sort type supported, "N" for natural order, "n" for number, "u" for user-defined order and "r" for reverse. e.g., "-k 1" or "-k A:r" or ""-k 1:nr -k 2" (default [1])
  -L, --levels strings   user-defined level file (one level per line, multiple values supported). format: <field>:<level-file>.  e.g., "-k name:u -L name:level.txt"

Examples

plot

Usage

plot common figures

Notes:

  1. Output file can be set by flag -o/--out-file.
  2. File format is determined by the out file suffix.
     Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff
  3. If flag -o/--out-file not set (default), image is written to stdout,
     you can display the image by pipping to "display" command of Imagemagic
     or just redirect to file.

Usage:
  csvtk plot [command]

Available Commands:
  box         plot boxplot
  hist        plot histogram
  line        line plot and scatter plot

Flags:
      --axis-width float     axis width (default 1.5)
  -f, --data-field string    column index or column name of data (default "1")
      --format string        image format for stdout when flag -o/--out-file not given. available values: eps, jpg|jpeg, pdf, png, svg, and tif|tiff. (default "png")
  -g, --group-field string   column index or column name of group
      --height float         Figure height (default 4.5)
      --label-size int       label font size (default 14)
      --tick-width float     axis tick width (default 1.5)
      --title string         Figure title
      --title-size int       title font size (default 16)
      --width float          Figure width (default 6)
      --x-max string         maximum value of X axis
      --x-min string         minimum value of X axis
      --xlab string          x label text
      --y-max string         maximum value of Y axis
      --y-min string         minimum value of Y axis
      --ylab string          y label text

Note that most of the flags of plot are global flags of the subcommands hist, box and line

Notes of image output

  1. Output file can be set by flag -o/--out-file.
  2. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff
  3. If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to "display" command of Imagemagic or just redirect to file.

plot hist

Usage

plot histogram

Notes:

  1. Output file can be set by flag -o/--out-file.
  2. File format is determined by the out file suffix.
     Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff
  3. If flag -o/--out-file not set (default), image is written to stdout,
     you can display the image by pipping to "display" command of Imagemagic
     or just redirect to file.

Usage:
  csvtk plot hist [flags]

Flags:
      --bins int          number of bins (default 50)
      --color-index int   color index, 1-7 (default 1)

Examples

plot box

Usage

plot boxplot

Notes:

  1. Output file can be set by flag -o/--out-file.
  2. File format is determined by the out file suffix.
     Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff
  3. If flag -o/--out-file not set (default), image is written to stdout,
     you can display the image by pipping to "display" command of Imagemagic
     or just redirect to file.

Usage:
  csvtk plot box [flags]

Flags:
      --box-width float   box width
      --horiz             horize box plot

Examples

plot line

Usage

line plot and scatter plot

Notes:

  1. Output file can be set by flag -o/--out-file.
  2. File format is determined by the out file suffix.
     Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff
  3. If flag -o/--out-file not set (default), image is written to stdout,
     you can display the image by pipping to "display" command of Imagemagic
     or just redirect to file.

Usage:
  csvtk plot line [flags]

Flags:
  -x, --data-field-x string   column index or column name of X for command line
  -y, --data-field-y string   column index or column name of Y for command line
      --legend-left           locate legend along the left edge of the plot
      --legend-top            locate legend along the top edge of the plot
      --line-width float      line width (default 1.5)
      --point-size float      point size (default 3)
      --scatter               only plot points

Examples

genautocomplete

Usage

generate shell autocompletion script

Note: The current version supports Bash only.
This should work for *nix systems with Bash installed.

Howto:

1. run: csvtk genautocomplete

2. create and edit ~/.bash_completion file if you don't have it.

        nano ~/.bash_completion

   add the following:

        for bcfile in ~/.bash_completion.d/* ; do
          . $bcfile
        done

Usage:
  csvtk genautocomplete [flags]

Flags:
      --file string   autocompletion file (default "/home/shenwei/.bash_completion.d/csvtk.sh")
  -h, --help          help for genautocomplete
      --type string   autocompletion type (currently only bash supported) (default "bash")