How to Bulk insert with a dynamic value for a column - php

Question

the situation is this:
I have 200 txt files with different names like 601776.txt each file's name is actually an ID_foo and it contains some data like this (2 columns):
04004 Albánchez
04006 Albox
04008 Alcóntar
04009 Alcudia de Monteagud
.
.
.
now I wanna BULK INSERT these TXT files into a SQL Server Table which has 3 column one of these columns should be the name of the txt file. I'm using a PHP script, so I made a loop to get the file names and then what?
BULK INSERT Employee_Table
FROM '../home/601776.txt'
WITH (
FIELDTERMINATOR ='\t',
ROWTERMINATOR = ''\n''
)
how can i set the third column while bulk inserting with $file_name variable in each loop?
Do you think it is a better idea if it is possible to insert the table by reading the txt file line by line? And how?
Thanks

Answer 1

This is one of the few times that a cursor is actually ideal in SQL Server. Here's a way. Once you see the PRINT statement and are satisfied you can comment it out and uncomment out the two lines below it. I put some logic in to add the file name and a processed date which is usually needed, but your table definition would need these columns. It should get the idea across.
---------------------------------------------------------------------------------------------------------------
--Set some variables
---------------------------------------------------------------------------------------------------------------
DECLARE #dt VARCHAR(10) --date variable but stored as VARCHAR for formatting of file name
DECLARE #fileLocation VARCHAR(128) = 'E:\DATA_TRANSFERS\' --production location which is \\issqlstd01 but the xp_dirtree didn't like this
DECLARE #sql NVARCHAR(4000) --dynamic sql variable
DECLARE #fileName VARCHAR(128) --full file name variable
---------------------------------------------------------------------------------------------------------------
--Get a list of all the file names in the directory
---------------------------------------------------------------------------------------------------------------
IF OBJECT_ID('tempdb..#FileNames') IS NOT NULL DROP TABLE #FileNames
CREATE TABLE #FileNames (
id int IDENTITY(1,1)
,subdirectory nvarchar(512)
,depth int
,isfile bit)
INSERT #FileNames (subdirectory,depth,isfile)
EXEC xp_dirtree #fileLocation, 1, 1
---------------------------------------------------------------------------------------------------------------
--Create a cursor to fetch the file names
---------------------------------------------------------------------------------------------------------------
DECLARE c CURSOR FOR
select subdirectory from #FileNames where isfile = 1
OPEN c
FETCH NEXT FROM c INTO #fileName
---------------------------------------------------------------------------------------------------------------
--For each file, bulk insert
---------------------------------------------------------------------------------------------------------------
WHILE ##FETCH_STATUS = 0
BEGIN
SET #sql = 'BULK INSERT Employee_Table FROM '''+ #fileLocation + #fileName +''' WITH (FIELDTERMINATOR = ''\t'',KEEPNULLS,ROWTERMINATOR = ''0x0a'')'
--Try the bulk insert, if error is thrown log the error
--Also update the Table Columns which aren't a part of the original file (load date and original file name)
BEGIN TRY
PRINT(#sql)
--EXEC(#sql)
--UPDATE Employee_Table SET OrigFile = #fileName, LoadDate = GETDATE() WHERE OrigFile IS NULL
END TRY
BEGIN CATCH
SELECT ERROR_MESSAGE()
END CATCH
FETCH NEXT FROM c INTO #fileName
END
CLOSE c
DEALLOCATE c
GO