473,289 Members | 2,087 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,289 software developers and data experts.

Simple script to merge multiple text files

Hi,

I have just started using Python and I am slowly getting into it. I wanted to make a little script to merge all files in a directory into one.

All of these files will be text files.

I know I need to use os.path.walk() to walk through all the files, and if not os.path.isdir(filename) with open(filename,'r').read() to read all the files that are not directories.

Can someone give me a clue or a very simple example that I can work with please?

Thanks in advance.
Nov 1 '07 #1
6 30538
bvdet
2,851 Expert Mod 2GB
Hi,

I have just started using Python and I am slowly getting into it. I wanted to make a little script to merge all files in a directory into one.

All of these files will be text files.

I know I need to use os.path.walk() to walk through all the files, and if not os.path.isdir(filename) with open(filename,'r').read() to read all the files that are not directories.

Can someone give me a clue or a very simple example that I can work with please?

Thanks in advance.
I think this code will do what you want:
Expand|Select|Wrap|Line Numbers
  1. import os
  2.  
  3. def dir_list(dir_name, subdir, *args):
  4.     '''Return a list of file names in directory 'dir_name'
  5.     If 'subdir' is True, recursively access subdirectories under 'dir_name'.
  6.     Additional arguments, if any, are file extensions to add to the list.
  7.     Example usage: fileList = dir_list(r'H:\TEMP', False, 'txt', 'py', 'dat', 'log', 'jpg')
  8.     '''
  9.     fileList = []
  10.     for file in os.listdir(dir_name):
  11.         dirfile = os.path.join(dir_name, file)
  12.         if os.path.isfile(dirfile):
  13.             if len(args) == 0:
  14.                 fileList.append(dirfile)
  15.             else:
  16.                 if os.path.splitext(dirfile)[1][1:] in args:
  17.                     fileList.append(dirfile)
  18.  
  19.         # recursively access file names in subdirectories
  20.         elif os.path.isdir(dirfile) and subdir:
  21.             # print "Accessing directory:", dirfile
  22.             fileList += dir_list(dirfile, subdir, *args)
  23.     return fileList
  24.  
  25. def combine_files(fileList, fn):
  26.     f = open(fn, 'w')
  27.     for file in fileList:
  28.         print 'Writing file %s' % file
  29.         f.write(open(file).read())
  30.     f.close()
  31.  
  32. if __name__ == '__main__':
  33.     search_dir = "C:/directory"
  34.     fn = "output_file.txt"
  35.     combine_files(dir_list(search_dir, False, 'txt'), fn)
Nov 1 '07 #2
Thanks , you have helped a lot :)
Nov 1 '07 #3
ghostdog74
511 Expert 256MB
another way
Expand|Select|Wrap|Line Numbers
  1. import os,shutil
  2. f=open("/tmp/fileappend.txt","a")
  3. for r,d,fi in os.walk("/home/me"):
  4.     for files in fi:
  5.         if files.endswith(".txt"):                         
  6.             g=open(os.path.join(r,files))
  7.             shutil.copyfileobj(g,f)
  8.             g.close()
  9. f.close()
  10.  
Nov 2 '07 #4
I think this code will do what you want:
Expand|Select|Wrap|Line Numbers
  1. import os
  2.  
  3. def dir_list(dir_name, subdir, *args):
  4.     '''Return a list of file names in directory 'dir_name'
  5.     If 'subdir' is True, recursively access subdirectories under 'dir_name'.
  6.     Additional arguments, if any, are file extensions to add to the list.
  7.     Example usage: fileList = dir_list(r'H:\TEMP', False, 'txt', 'py', 'dat', 'log', 'jpg')
  8.     '''
  9.     fileList = []
  10.     for file in os.listdir(dir_name):
  11.         dirfile = os.path.join(dir_name, file)
  12.         if os.path.isfile(dirfile):
  13.             if len(args) == 0:
  14.                 fileList.append(dirfile)
  15.             else:
  16.                 if os.path.splitext(dirfile)[1][1:] in args:
  17.                     fileList.append(dirfile)
  18.  
  19.         # recursively access file names in subdirectories
  20.         elif os.path.isdir(dirfile) and subdir:
  21.             # print "Accessing directory:", dirfile
  22.             fileList += dir_list(dirfile, subdir, *args)
  23.     return fileList
  24.  
  25. def combine_files(fileList, fn):
  26.     f = open(fn, 'w')
  27.     for file in fileList:
  28.         print 'Writing file %s' % file
  29.         f.write(open(file).read())
  30.     f.close()
  31.  
  32. if __name__ == '__main__':
  33.     search_dir = "C:/directory"
  34.     fn = "output_file.txt"
  35.     combine_files(dir_list(search_dir, False, 'txt'), fn)
This seemed to work for a while for me now it comes up with "NameError: name 'False' is not defined"
Nov 2 '07 #5
bvdet
2,851 Expert Mod 2GB
This seemed to work for a while for me now it comes up with "NameError: name 'False' is not defined"
Expand|Select|Wrap|Line Numbers
  1. >>> bool(0)
  2. False
  3. >>> bool(1)
  4. True
  5. >>> 
'0' and '1' can be substituted for 'False' and 'True' respectively. The question is what happened to 'False' on your system?
Nov 2 '07 #6
Expand|Select|Wrap|Line Numbers
  1. >>> bool(0)
  2. False
  3. >>> bool(1)
  4. True
  5. >>> 
'0' and '1' can be substituted for 'False' and 'True' respectively. The question is what happened to 'False' on your system?
Did a quick reset on my machine and is working now. very strange.
Nov 9 '07 #7

Sign in to post your reply or Sign up for a free account.

Similar topics

0
by: Guillaume Lahitette | last post by:
Hello everyone, I am looking for a lightweight Java Applet to diff and merge two text files. The functionality would be a subset of what WinMerge (http://winmerge.sourceforge.net) offers: -...
1
by: Charlie | last post by:
Hello, I have data in an Access table that I would like to export to multiple HTML tables. I would like to split the data in the Access table (about 92,000 records) into multiple HTML...
2
by: Chris Murphy via DotNetMonster.com | last post by:
Hey all, just wondering if anyone can point me in the right direction. I'm developing a solution that allows a user to store multiple text-based content (like code snippets, notes, documents etc.)...
5
by: Buddhist[CHinA] | last post by:
The text files are not only the .txt files, but also all ascii files. Thx.
9
by: malla | last post by:
Hi, I am trying to open a bunch of files and get data from them into one single file. I am sure this process can be automated if I name the files that I want to open in a regular pattern (say...
2
by: ManningFan | last post by:
I'm running through a database in a loop, and each time the loop finishes I would like to export the values of some variables to new text files. In reality it's more complex than this, but this is...
3
emaghero
by: emaghero | last post by:
Hello all, I want to open multiple txt files with similar names in C++ I have attempted this with the following code //Create as many txt files as there are valid propagation constants...
2
by: as001 | last post by:
Hi, I'm writing a windows application in C# using VS 2003. I got stuck where it has to write multiple output text files. Here's my piece of code: for loop { Random r = new Random();...
5
by: maral | last post by:
Hi every one, this is my first post here! I'm using GATE toolkit for information retrieval and text analysis, but i really need java for some parts. I have managed to find a specific word in...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.