Data Import and Submission Export Tutorial

This tutorial will use R to import the data and output a sample submission.

To begin, make sure your R session has its working directory set to the same directory where your data is located. To view your current working directory, run the command getwd() in the R console. Use the R options File -> Change dir... in the RGui to set your working directory or use the setwd('insert/your/path/to/data.csv') command to set your working directory.

To double check that train.csv and test.csv are in your current working directory, the following command should return TRUE twice, as seen in the output below.

c('train.csv', 'test.csv') %in% list.files()

1. TRUE
2. TRUE

Data Import

Now that we have an R session with our two data files in the working directory, read the comma separated data using the read.csv() function and view a few predictor summaries.

train = read.csv('train.csv')

head(train)

	RowID	CalendarYear	ModelYear	Make	Model	Cat1	Cat2	Cat3	Cat4	Cat5	⋯	Var5	Var6	Var7	Var8	NVCat	NVVar1	NVVar2	NVVar3	NVVar4	Response
1	418079	2005	2004	AU	AU.14	B	C	A	A	A		0.02168622	-0.6852558	-0.5912954	-0.2584976	F	-0.2315299	-0.2661168	4.209404	-0.2514189	0
2	232625	2006	2003	R	R.30	B	C	B	A	A		-0.3529838	-0.4232408	-0.6281582	0.05436843	O	-0.2315299	-0.2661168	-0.2723372	-0.2514189	0
3	379029	2006	2006	AU	AU.14	B	A	A	A	A		0.06926337	-0.6852558	-0.5912954	-0.163872	M	-0.2315299	-0.2661168	-0.2723372	-0.2514189	1
4	181458	2007	2000	BU	BU.38	F	C	A	C	A		0.0871048	-0.1717531	-0.9722114	0.2064257	O	-0.2315299	-0.2661168	-0.2723372	-0.2514189	0
5	192434	2005	1999	BU	BU.38	F	A	A	C	A		0.0871048	-0.2723481	-0.9722114	0.290987	M	-0.2315299	-0.2661168	-0.2723372	-0.2514189	1
6	443321	2007	2005	AU	AU.11	B	C	B	A	A		0.1406291	-0.1635651	-0.5421449	0.5499219	O	-0.2315299	-0.2661168	-0.2723372	-0.2514189	0

hist(train$Var4, main = "Histogram of Var4", xlab = "Var4")

summary(train$Var8)

    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
-1.48500 -0.52320 -0.24210 -0.00946  0.16210 33.90000 

Also read the test set into your R session via the read.csv() function.

test = read.csv('test.csv')

head(test)

	RowID	CalendarYear	ModelYear	Make	Model	Cat1	Cat2	Cat3	Cat4	Cat5	⋯	Var4	Var5	Var6	Var7	Var8	NVCat	NVVar1	NVVar2	NVVar3	NVVar4
1	491169	2009	2006	AN	AN.4	B	B	B	A	A		-0.9014148	-0.7871253	-0.939083	-0.8124724	-0.7338132	J	-0.2315299	2.021183	-0.2723372	1.408637
2	661907	2009	2001	BU	BU.5	B	C	B	A	A		1.043661	0.3963563	0.004873184	-0.82476	0.808638	O	-0.2315299	-0.2661168	-0.2723372	-0.2514189
3	489459	2009	2005	Y	Y.21	B	C	A	A	A		-0.7830189	-0.6443939	-0.8408273	-0.6895963	-0.8164055	E	-0.2315299	-0.2661168	-0.2723372	-0.2514189
4	496824	2009	2006	BF	BF.18	B	C	A	A	A		0.9929196	0.5331406	0.2493426	-0.5544326	0.01006398	O	-0.2315299	-0.2661168	-0.2723372	-0.2514189
5	464567	2009	2008	K	K.40	E	C	A	A	A		-1.104379	-1.179637	-1.061903	-0.1366537	-0.616215	O	-0.2315299	-0.2661168	-0.2723372	-0.2514189
6	307296	2009	2007	K	K.40	E	C	A	A	A		-1.104379	-1.179637	-1.061903	-0.2595298	-0.685133	L	-0.2315299	-0.2661168	-0.2723372	-0.2514189

plot(density(train$Var2), main = "Density of Var2")

dim(train)

1. 100000
2. 32

dim(test)

1. 40000
2. 31

The training and testing sets have a different number of columns. This is, of course, because the test set does not contain the response variable. The following command will tell us which column is contained in the training set and not in the testing set.

setdiff(names(train), names(test))

"Response"

Data Export

When making a submission, the predictions need to be exported in a certain fashion. The example below will generate random uniform numbers and use them as our predictions.

numberOfObservationsInTestSet = nrow(test)
vectorOfPredictions = runif(numberOfObservationsInTestSet, 0, 1)
summary(vectorOfPredictions)

     Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
0.0000013 0.2508000 0.4968000 0.4987000 0.7473000 0.9999000 

outputDataSet = data.frame("RowID" = test$RowID,
                           "ProbabilityOfResponse" = vectorOfPredictions)

Inspect data set before export

head(outputDataSet)

	RowID	ProbabilityOfResponse
1	491169	0.5298258
2	661907	0.7990218
3	489459	0.42184
4	496824	0.7455896
5	464567	0.1310719
6	307296	0.8185932

The following command will output a comma separated file to the current working directory. Find your current working directory again by executing the getwd() command.

write.csv(outputDataSet, "submissionExample.csv", row.names = FALSE)