2
1

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

DictionaryのTValueにStreamWriterを入れる

Last updated at Posted at 2017-09-18

メモリ4GB(32bit)の環境で、2.5GBのCSVを処理する必要に迫られました。
当初は1行ごとにStreamWriterを開く、書く閉じるを繰り返していましたが、あまりに遅かったため全部開きっぱなしにて効率よく管理するために、Dictionaryに持たせてみました。
想定通り動いてくれたのでメモ書き。

Program.cs
using System;
using System.Collections.Generic;
using System.IO;

namespace ConsoleApp2
{
    class Program
    {
        static void Main(string[] args)
        {
            string path = @"C:\test\";
            int div = 200;

            // テスト用ファイルを作る
            using (StreamWriter sw = new StreamWriter(path + "test.txt"))
            {
                for (int i = 1; i < 50000000; i++)
                {
                    sw.WriteLine(i.ToString() + ",いろはにほへとちりぬるをわかよたれそつねならむうゐのおくやまけふこえてあさきゆめみしゑひもせす");
                }
            }

            // 書き込み用のStreamWriterを開く
            Dictionary<int, StreamWriter> writer = new Dictionary<int, StreamWriter>();
            for (int i = 0; i < div; i++)
            {
                writer.Add(i, new StreamWriter(path + i.ToString() + ".txt"));
            }

            // ここで内容判断、書き込み先の分岐処理を行う
            using (StreamReader sr = new StreamReader(path + "test.txt"))
            {
                while (sr.Peek() >= 0)
                {
                    string s = sr.ReadLine();
                    writer[Convert.ToInt32(s.Split(',')[0]) % div].WriteLine(s);
                }
            }

            // 忘れずにStreamWriterを閉じる
            for (int i = 0; i < div; i++)
            {
                writer[i].Close();
                writer[i].Dispose();
            }
        }
    }
}
2
1
1

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
2
1

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?