Error: EISDIR: illegal operation on a directory

[webframp]

Hey there,

Neat looking project here since VPC Flow logs are such a pain usually to search through.

I tried this out today and generate-graph returns this error:
(note: I sanitized the account number)

Error: EISDIR: illegal operation on a directory, open 'data/AWSLogs/${account_number}/'

It does create the directory structure though:

❯ tree data/AWSLogs/
data/AWSLogs/
└── ${account_number}
    └── vpcflowlogs
        └── us-east-1

I tried the suggestion from #2 to modify createFolderIfNotExists but it made no difference any ideas or suggestions for debugging?

thanks!

[FlorianPfisterer]

Hi! Thanks for the feedback.
Can you specify in which line this error is specifically thrown?

[webframp]

Hey, sorry for the delay. I'd be happy to provide that, how do you suggest I get it for you? node isn't really my area, so I'm not sure how to step through the execution like I would with Go or another language debugger.

[vlebo]

Hey @FlorianPfisterer i also experienced the same issue, after some debugging with @ivanblagdan managed to make it work. Seems be the issue with some of the log files in the bucket

diff --git a/graph-generator/s3-download.ts b/graph-generator/s3-download.ts
index c471e93..eb7c2c6 100644
--- a/graph-generator/s3-download.ts
+++ b/graph-generator/s3-download.ts
@@ -44,7 +44,11 @@ const downloadLogs = async (): Promise<string[]> => {
                 if (err) reject(err);
                 else resolve(fullPath);
             });
-        });
+        }).catch((err)=>{
+//             console.log(err)
+//             console.log(data)
+//             console.log(object)
+                       });
     })) as string[];
 
     return filePaths;
@@ -53,7 +57,7 @@ const downloadLogs = async (): Promise<string[]> => {
 const unzipLogs = async (filePaths: string[]): Promise<string[]> => {
     return removeUndef(await Promise.all(filePaths.map(filePath => {
         return new Promise<string | undefined>((resolve, reject) => {
-            if (filePath.endsWith('.gz')) {
+            if (filePath !== undefined && filePath.endsWith('.gz')) {
                 const newFileName = filePath.substr(0, filePath.length - '.gz'.length);
                 gunzip(filePath, newFileName, () => {
                     // now remove the file

Pretty sure there is a better way to fix this but I needed it for a single run so this hack worked for me :)

When running w/o --transpile-only flag error was

ts-node ./graph-generator/index.ts
/home/vlebo/.nvm/versions/node/v14.17.6/lib/node_modules/ts-node/src/index.ts:692
    return new TSError(diagnosticText, diagnosticCodes);
           ^
TSError: ⨯ Unable to compile TypeScript:
graph-generator/s3-download.ts:35:27 - error TS2769: No overload matches this call.
  Overload 1 of 2, '(params: ListObjectsRequest, callback?: ((err: AWSError, data: ListObjectsOutput) => void) | undefined): Request<ListObjectsOutput, AWSError>', gave the following error.
    Type 'string | undefined' is not assignable to type 'string'.
      Type 'undefined' is not assignable to type 'string'.
  Overload 2 of 2, '(callback?: ((err: AWSError, data: ListObjectsOutput) => void) | undefined): Request<ListObjectsOutput, AWSError>', gave the following error.
    Argument of type '{ Bucket: string | undefined; }' is not assignable to parameter of type '(err: AWSError, data: ListObjectsOutput) => void'.
      Object literal may only specify known properties, and 'Bucket' does not exist in type '(err: AWSError, data: ListObjectsOutput) => void'.

35     const objects = await s3.listObjects({ Bucket: bucket }).promise();
                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  node_modules/aws-sdk/clients/s3.d.ts:3217:5
    3217     Bucket: BucketName;
             ~~~~~~
    The expected type comes from property 'Bucket' which is declared here on type 'ListObjectsRequest'
graph-generator/s3-download.ts:38:43 - error TS2769: No overload matches this call.
  Overload 2 of 2, '(callback?: ((err: AWSError, data: GetObjectOutput) => void) | undefined): Request<GetObjectOutput, AWSError>', gave the following error.
    Argument of type '{ Key: string | undefined; Bucket: string | undefined; }' is not assignable to parameter of type '(err: AWSError, data: GetObjectOutput) => void'.
      Object literal may only specify known properties, and 'Key' does not exist in type '(err: AWSError, data: GetObjectOutput) => void'.

38         const data = await s3.getObject({ Key: object.Key, Bucket: bucket }).promise();
                                             ~~~~~~~~~~~~~~~


    at createTSError (/home/vlebo/.nvm/versions/node/v14.17.6/lib/node_modules/ts-node/src/index.ts:692:12)
    at reportTSError (/home/vlebo/.nvm/versions/node/v14.17.6/lib/node_modules/ts-node/src/index.ts:696:19)
    at getOutput (/home/vlebo/.nvm/versions/node/v14.17.6/lib/node_modules/ts-node/src/index.ts:883:36)
    at Object.compile (/home/vlebo/.nvm/versions/node/v14.17.6/lib/node_modules/ts-node/src/index.ts:1185:30)
    at Module.m._compile (/home/vlebo/.nvm/versions/node/v14.17.6/lib/node_modules/ts-node/src/index.ts:1309:30)
    at Module._extensions..js (internal/modules/cjs/loader.js:1101:10)
    at Object.require.extensions.<computed> [as .ts] (/home/vlebo/.nvm/versions/node/v14.17.6/lib/node_modules/ts-node/src/index.ts:1313:12)
    at Module.load (internal/modules/cjs/loader.js:937:32)
    at Function.Module._load (internal/modules/cjs/loader.js:778:12)
    at Module.require (internal/modules/cjs/loader.js:961:19) {
  diagnosticText: '\x1B[96mgraph-generator/s3-download.ts\x1B[0m:\x1B[93m35\x1B[0m:\x1B[93m27\x1B[0m - \x1B[91merror\x1B[0m\x1B[90m TS2769: \x1B[0mNo overload matches this call.\n' +
    "  Overload 1 of 2, '(params: ListObjectsRequest, callback?: ((err: AWSError, data: ListObjectsOutput) => void) | undefined): Request<ListObjectsOutput, AWSError>', gave the following error.\n" +
    "    Type 'string | undefined' is not assignable to type 'string'.\n" +
    "      Type 'undefined' is not assignable to type 'string'.\n" +
    "  Overload 2 of 2, '(callback?: ((err: AWSError, data: ListObjectsOutput) => void) | undefined): Request<ListObjectsOutput, AWSError>', gave the following error.\n" +
    "    Argument of type '{ Bucket: string | undefined; }' is not assignable to parameter of type '(err: AWSError, data: ListObjectsOutput) => void'.\n" +
    "      Object literal may only specify known properties, and 'Bucket' does not exist in type '(err: AWSError, data: ListObjectsOutput) => void'.\n" +
    '\n' +
    '\x1B[7m35\x1B[0m     const objects = await s3.listObjects({ Bucket: bucket }).promise();\n' +
    '\x1B[7m  \x1B[0m \x1B[91m                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\x1B[0m\n' +
    '\n' +
    '  \x1B[96mnode_modules/aws-sdk/clients/s3.d.ts\x1B[0m:\x1B[93m3217\x1B[0m:\x1B[93m5\x1B[0m\n' +
    '    \x1B[7m3217\x1B[0m     Bucket: BucketName;\n' +
    '    \x1B[7m    \x1B[0m \x1B[96m    ~~~~~~\x1B[0m\n' +
    "    The expected type comes from property 'Bucket' which is declared here on type 'ListObjectsRequest'\n" +
    '\x1B[96mgraph-generator/s3-download.ts\x1B[0m:\x1B[93m38\x1B[0m:\x1B[93m43\x1B[0m - \x1B[91merror\x1B[0m\x1B[90m TS2769: \x1B[0mNo overload matches this call.\n' +
    "  Overload 2 of 2, '(callback?: ((err: AWSError, data: GetObjectOutput) => void) | undefined): Request<GetObjectOutput, AWSError>', gave the following error.\n" +
    "    Argument of type '{ Key: string | undefined; Bucket: string | undefined; }' is not assignable to parameter of type '(err: AWSError, data: GetObjectOutput) => void'.\n" +
    "      Object literal may only specify known properties, and 'Key' does not exist in type '(err: AWSError, data: GetObjectOutput) => void'.\n" +
    '\n' +
    '\x1B[7m38\x1B[0m         const data = await s3.getObject({ Key: object.Key, Bucket: bucket }).promise();\n' +
    '\x1B[7m  \x1B[0m \x1B[91m                                          ~~~~~~~~~~~~~~~\x1B[0m\n' +
    '\n',
  diagnosticCodes: [ 2769, 2769 ]
}

[jpfreeley]

+1 same issue .. but the fix that @vlebo seems to help

[brc]

Thank you for patch, @vlebo. Did the trick. 👍

@FlorianPfisterer Thank you for sharing the project. Here is the error with npm v8.4.1:

[vpc-flow-log-analysis]$ npm run generate-graph

> [email protected] generate-graph
> ts-node ./graph-generator/index.ts

Error: EISDIR: illegal operation on a directory, open 'data/AWSLogs/123456789000/'

[vpc-flow-log-analysis]$

There was no logs/ directory afterwards.
data/ had an empty subdirectory; looked like this:

[vpc-flow-log-analysis]$ find data/
data/
data/AWSLogs
data/AWSLogs/123456789000
[vpc-flow-log-analysis]$

HTH.

[FlorianPfisterer]

I see. Maybe they changed the format of these logs. Thanks for the feedback & help @vlebo / @invsblduck.
Feel free to submit a PR with your fix!

[vlebo]

I am pretty sure there is a better way of fixing it and PR-ing then my hack (i am not a developer) :)