convert-embeddings-format

Overview¶

The convert-embeddings-command converts a KGTK edge file to word2vec format or Google Projector files format. Currently, only these two file formats are supported, more file formats will be added in a later revision of the command.

The vectors produced by the text-embeddings and the graph-embeddings commands are stored as ,(comma) separated string in node2.

Word2vec Format¶

The word2vec is a text file, where the first line has the number of vectors in the file and the dimension of each vector.

There is one line per entity, with entity id and the vector components separated by space.

Google Projector File(s) Format¶

Google Projector allows users to visualize high dimensional vectors. Google projector requires two files, an embeddings file and a metadata file.

The embeddings file contains vectors for each entity, with vector components separated by tabs.

The metadata file contains information about each entity like label, description, counts etc. The order of entities in the metadata file should be same as the embeddings file.

Usage¶

usage: kgtk convert-embeddings-format [-h] [-i INPUT_FILE]
                                      [--node-file NODE_FILE] [-o OUTPUT_FILE]
                                      [--metadata-file METADATA_FILE]
                                      [--input-property INPUT_PROPERTY]
                                      [--output-format OUTPUT_FORMAT]
                                      [--metadata-columns METADATA_COLUMNS]

Converts KGTK edge embeddings file to word2vec or Google Projector format.Takes an optional node file for Google Project format to create a metadata file. Processes only top 10,000 rows from the edge file for Google Projector as it only accepts 10,000 rows.
Additional options are shown in expert help.
kgtk --expert convert-embeddings-format --help

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT_FILE, --input-file INPUT_FILE
                        KGTK input files (May be omitted or '-' for stdin.)
  --node-file NODE_FILE
                        The KGTK node file for creating Google Projector
                        Metadata. All the columns in the node file will be
                        added to the metadata file by default. You can
                        customise this with the --metadata-columns option.
                        (Optional)
  -o OUTPUT_FILE, --output-file OUTPUT_FILE
                        The KGTK output file. (May be omitted or '-' for
                        stdout.)
  --metadata-file METADATA_FILE
                        The output metadata file for Google Projector. If
                        --output-format == gprojector and--metadata-file is
                        not specified, a file named
                        `kgtk_embeddings_gprojector_metadata.tsv` will be
                        created in USER_HOME (Optional)
  --input-property INPUT_PROPERTY
                        The property name for embeddings in the input KGTK
                        edge file. (default=embeddings).
  --output-format OUTPUT_FORMAT
                        The desired output file format: word2vec|gprojector
                        (default=word2vec)
  --metadata-columns METADATA_COLUMNS
                        A comma separated string of columns names in the input
                        file or the --node-file to be used for creating the
                        metadata file for Google projector. Only to be used
                        when --output-format == 'gprojector'. If --node-file
                        is specified, the command will look for --metadata-
                        columns in the --node-file, otherwise input file.The
                        command will throw an error if the columns specified
                        are not in either of the files.

Examples¶

Convert KGTK to word2vec format¶

kgtk convert-embeddings-format -i examples/doc/convert_embeddings_edge.tsv --input-property graph_embeddings -o embeddings_word2vec.txt

Let's look at the output word2vec file ,

>>>head embeddings_word2vec.txt

19 30
Q494335 -0.162911773 0.071842454 -0.223435551 -0.289004564 0.834948838 ...
Q1278301 0.039679553 -0.115788229 -0.179974616 0.590080559 0.158913493 ...
Q611586 -0.015744781 -0.020170633 -0.313573331 0.515067458 0.039014913 ...
Q816369 0.488982528 -0.719077468 0.109514274 0.301486224 -0.402110636 ...
Q26833575 0.095825180 -0.207610607 -0.293776900 0.226735979 0.113529690 ...

Convert KGTK to gprojector format, use all columns in the node file for metadata¶

kgtk convert-embeddings-format \
  -i examples/doc/convert_embeddings_edge.tsv  \
  --node-file examples/doc/convert_embeddings_node.tsv \
  --output-format gprojector \
  --input-property graph_embeddings \
  --metadata-file gprojector_metadata.tsv \
  -o embeddings_gprojector.tsv

Let's take a look at the metadata and embeddings file ,

>>> head gprojector_metadata.tsv

id	label	type	type_label
Q494335	Tours University	Q3551775	university in France
Q1278301	Robert Bouline	Q5	human
Q611586	William Monahan	Q5	human
Q816369	rated voltage	Q25428	voltage

>>> head embeddings_gprojector.tsv

-0.162911773	0.071842454	-0.223435551	-0.289004564	0.834948838	-0.373376131	1.436196566	-0.942946911	...
0.039679553	-0.115788229	-0.179974616	0.590080559	0.158913493	0.008464743	0.712676883	0.380636603	...
-0.015744781	-0.020170633	-0.313573331	0.515067458	0.039014913	-0.114478707	0.770638645	0.304640383	...
0.488982528	-0.719077468	0.109514274	0.301486224	-0.402110636	0.291337997	0.829619348	-0.365474463	...
0.095825180	-0.207610607	-0.293776900	0.226735979	0.113529690	0.536592960	0.747583449	-0.221452087	...

Convert KGTK to gprojector format, use columns: `label` and `type_label` in the node file for metadata¶

kgtk convert-embeddings-format \
  -i examples/doc/convert_embeddings_edge.tsv  \
  --node-file examples/doc/convert_embeddings_node.tsv \
  --metadata-columns label,type_label \
  --output-format gprojector \
  --input-property graph_embeddings \
  --metadata-file gprojector_metadata.tsv \
  -o embeddings_gprojector.tsv

Let's take a look at the customized metadata file

>>> head gprojector_metadata.tsv

label	type_label
Tours University	university in France
Robert Bouline	human
William Monahan	human
rated voltage	voltage
France-Guernsey border	international border

Convert KGTK to gprojector format, use columns: `node1_label` and `type` in the edge file for metadata¶

kgtk convert-embeddings-format \
  -i examples/doc/convert_embeddings_edge.tsv  \
  --metadata-columns node1_label,type \
  --output-format gprojector \
  --input-property graph_embeddings \
  --metadata-file gprojector_metadata.tsv \
  -o embeddings_gprojector.tsv