+ Demo + · + Deploying Guide + · + FAQ +
**Development will be paused for a few weeks, I will resume the project as soon as possible.** ## What is this? + `next-gdrive-index` is an indexer for Google Drive, it's a simple project that I made to index my files in Google Drive. It's aim to simplify the process of sharing files using Google Drive, and also implements some features that I think is useful for sharing files. This project are **HEAVILY INSPIRED** by [onedrive-vercel-index](https://github.com/spencerwooo/onedrive-vercel-index) by SpencerWooo. ### Why I made this? -I know there already some Google Drive indexer out there, but all of them are using **Cloudflare** worker. -I've tried to use it, but the speed isn't that great for me. -Currently I'm using SpencerWooo's onedrive index, and it's working perfectly for me from usage perspective and speed. -But, if I compare the price between these two services, Google Drive still cheaper than Onedrive. +when I compare the price between these two services, Google Drive still cheaper than Onedrive. For 100GB plan, Google Drive is IDR 269.000 / ~$18 USD annually, and Onedrive is IDR 319.000 / ~$21 USD annually. For free plan, Google Drive offering 15GB of storage, and Onedrive offering 5GB of storage. I know there are a lot of people selling cheap edu account for Google Drive and Onedrive, but most of the time those account doesn't last long, especially Google Drive account. So I want to do it legit without buying cheap edu account or anything similar this time. ## Features -### Navigate through your Google Drive files and folders. -You can navigate through your Google Drive files and folders, just like you're browsing your local files. -You can also set the root folder, so you can share only specific folder in your Google Drive. -### Theme support. -Choose your side! +- **Private Index**, lock the whole index site with password. +- **Folder and file protection**, you can protect your folder and files with password. +- **Readme file**, you can add readme file for your folder, so you can add some description for your folder. +- **File preview**, you can preview some file types directly from the browser, without downloading it first. + - Image preview + - Video preview + - Audio preview + - Markdown preview + - PDF preview + - Document preview (docx, pptx, xlsx) + - Code preview + - Text preview + - Manga preview (cbz) +- **File search**, you can search for files and folders easily. +- **Direct download**, you can download files directly from index site without opening Google Drive. +- **Raw file link**, you can use the raw file link for hosting your web project assets or comments on forum. -### Site wide password protection. -You can lock the site so only those who know the password can access it. -NOTE: This only protect the site, not the files. The files are still accessible if you know the link. +## File Security -### Search for files and folders. -We implement native search for files and folders, so you can search for files and folders easily. +All files fetched from Google Drive **NEED TO BE SHARED** with `Anyone with the link can view` permission. +This is because Google Drive API can only access files that are shared with `Anyone with the link can view` permission. -### File preview for some file types. -You can preview some file types directly from the browser, without downloading it first. -Most files are Images, Videos, Audio, Markdown, and PDF. (I'm still adding more file types) +But, every files `id` and `webContentLink` are encrypted with `AES-256-CBC` using your own key, so no one can access your files without the key. +Except, if the files are larger than the `fileSizeLimit` (default: 4MB for vercel), then the download will be redirected to the raw file link. -### Direct download for files. -Download files directly from index site without opening Google Drive. +## Known Issue -### Raw file link for files. -You can use the raw file link for hosting your web project assets or comments on forum. +### File size limit -### Embed your readme file for folder. -You can embed your readme file for folder, so you can add some description for your folder. -Simply upload `.readme.md` on your folder, and it will be displayed on the folder page. +File size limit causing some files can't be previewed, and the download will be redirected to the raw file link. -### Override folder OpenGraph image. -You can override the folder OpenGraph image by uploading `.banner.(png/jpg/jpeg/webp/svg)` on your folder. +### Long Response Time -## Credits -- [onedrive-vercel-index](https://github.com/spencerwooo/onedrive-vercel-index) by SpencerWooo for the inspiration and also some file type helpers. +The flow of the project are: ---- +- Validate the path is valid or not. (Only applied for `/...[path]`) +- Fetch the password, readme, and files from Google Drive. +- Show to the user. -## Known Issues -- ~~Fetching files from Google Drive API take too much time, it triggers 504 Error on Vercel.~~ -- ~~Can't download big files, because of Vercel's 4.5MB limit for serverless function.~~ (Implement redirect to Google Drive download link instead of using Vercel serverless function for files bigger than 4MB) - - Need to set the folder to public, so it can be downloaded directly from Google Drive. - - The way to get files are changed because of this, now it's using the file name, and 8 first character of file id for hiding the id so no one can access it via Google Drive. - - All id and webContentLink are now encrypted using AES-128-CBC. -- Protected folder cause long response time, because it need to fetch the file / folder first > fetch each of the parents till root > check if it's inside protected folder > then checking the password. -- Download might show scan warning, because it's using Google Drive's direct download link. +So, if you have a lot of files in your Google Drive, it might take a long time to fetch all the files. +To improve the response time, I added a cache to the response, so the next time you access the same path, it will be faster. -## TODO -Todo list are moved to [here](./TODO.md). +ATM, it roughly take around 600 - 2 seconds to fetch all the data on my Google Drive. + +### No support for Google Shared Drive + +I don't have Google Shared Drive, so I can't test it and implement it. + +### No support for Google Docs, Sheets, and Slides + +For now, I don't have any plan to implement this, because I think it's not necessary. +All Google Drive files like Docs, Sheets, and Slides are hidden from the list. diff --git a/TODO.md b/TODO.md index 7776533..3f02df2 100644 --- a/TODO.md +++ b/TODO.md @@ -1,109 +1,41 @@ -# Todo -Detailed what I need to do for this project. -This is just a reminder for me, so I don't forget what I need to do. +- Redo how to handle error -Probably also include some ideas that I want to implement in the future. +# / +- ~~Override opengraph using banner image~~ +- ~~Render readme file~~ +- ~~Copy folder url~~ +- ~~Copy file direct url~~ +- ~~Download file url~~ +- ~~Index password~~ -## Frontend -### General -- ~~Site wide password protection~~ -- Migrate to app directory (?) +# /:path +- ~~Override opengraph using banner image / thumbnail image~~ +- ~~Render readme file~~ +- ~~Copy folder url~~ +- ~~Copy file direct url~~ +- ~~Download file url~~ +- ~~Password protected~~ + - Issues: Return 500 error if it's wrong password +- File Preview + - 3D > 3DPreview + - Audio > AudioPreview + - Archive > UnknownPreview + - RichText > MarkdownPreview + - OfficeWord > OfficePreview + - OfficeExcel > OfficePreview + - OfficePowerPoint > OfficePreview + - PDF > PDFPreview + - Database > UnknownPreview + - Image > ImagePreview + - Code > CodePreview + - Text > TextPreview + - Video > VideoPreview + - Font > UnknownPreview + - Default > UnknownPreview + - Binary > UnknownPreview -### Navbar `8/8 Completed` -- ~~Search for files~~ - - ~~Open / Close search modal~~ - - ~~Search for files or folder~~ - - ~~Configurable search limit~~ - - ~~Close search modal when click outside of modal~~ -- ~~Theme~~ - - ~~Switch theme~~ - - ~~Store theme preference in local storage~~ -- ~~Navbar menu~~ - -### Layout Container `10/10 Completed` -This part contains only breadcrumb and view selection. - -- ~~Breadcrumb~~ - - ~~Fetch breadcrumb~~ - - ~~Navigate through breadcrumb~~ - - ~~Limit breadcrumb~~ - - ~~Add `...` if breadcrumb is too long~~ - - ~~Add ellipsis if current file name is too long~~ - - ~~Configurable breadcrumb limit~~ -- ~~Layout view~~ - - ~~Grid view~~ - - ~~List view~~ - - ~~Store view preference in local storage~~ - - ~~Close view selection when click outside~~ - -### Home `/` | `4/5 Completed` -- ~~Fetch files~~ -- ~~Fetch readme if exist~~ -- ~~Fetch banner if exist~~ -- ~~Pagination~~ -- Protected folder - -### Folder `/folder/:id` | `5/6 Completed` -- ~~Check if folder exist~~ -- ~~Fetch files~~ -- ~~Fetch readme if exist~~ -- ~~Fetch banner if exist~~ -- ~~Pagination~~ -- Protected folder - -### File `/file/:id` | `` - -## Backend -### General -Since Google Drive direct download need the file to be public, I implement the `partial file id` to keep people from directly accessing the Google Drive itself, and also encrypt the `id` and `webContentLink` from Google Drive API to ensure that no one can access the file directly from Google Drive. - -- ~~Change `fileId` to `{file name}:{partial file id}`~~ -- ~~Encrypt `id` and `webContentLink`~~ -- ~~Making sure response time is fast enough. Aim for max 1.5s.~~ - - ~~Fetch root folder `600~900ms`~~ - - ~~Fetch file or folder `700~1400ms`~~ - - ~~Fetch breadcrumb `900~1500ms`~~ - - ~~Fetch readme `1100~1200ms`~~ - - ~~Fetch readme from specific folder `1200~1500ms`~~ - - ~~Fetch banner `500-800ms`~~ - - ~~Fetch banner from specific folder `500~800ms`~~ - - Protect folder - -### Fetch files `/api/files` | `16/17 Completed` - -- ~~Fetch file~~ -- ~~Fetch folder~~ -- ~~Paginated~~ -- ~~Fetch info regarding readme file~~ -- ~~Fetch info regarding banner file~~ -- ~~Fetch info regarding password file~~ (Not used yet) -- Download file - - ~~Redirect to Google Drive download link instead of using Vercel serverless function for files bigger than 4MB~~ - - ~~Only allow for files~~ - - Files inside protected folder -- ~~Get thumbnail for files~~ -- ~~Get banner inside folder~~ - - ~~Only if banner file is inside the folder~~ - - ~~Redirect to Google Drive download link instead of using Vercel serverless function for files bigger than 4MB~~ - - ~~Check if banner file is image~~ -- ~~Generate breadcrumb~~ - - ~~On folder~~ - - ~~On file~~ - - ~~Hide root folder~~ - - ~~Limiting breadcrumb length~~ - -### Get readme data `/api/readme` | `2/2 Completed` -This API route doesn't need authorization, because it's only used for getting readme data. - -- ~~Get root folder readme~~ -- ~~Get specific folder readme~~ - -### Get banner file id `/api/banner` | `2/2 Completed` -This API route doesn't need authorization, because it's only used for getting banner file id for Opengraph Image. - -- ~~Get root folder banner~~ -- ~~Get specific folder banner~~ - -### Search for files `/api/search` | `2/2 Completed` -- ~~Search for files and folder~~ -- ~~Limit search result~~ \ No newline at end of file +# /download/:path +- ~~Download file based on path~~ +- ~~Validate path~~ +- Protected file +- Token based for protected file \ No newline at end of file diff --git a/data/mockFiles.ts b/data/mockFiles.ts new file mode 100644 index 0000000..6b435f9 --- /dev/null +++ b/data/mockFiles.ts @@ -0,0 +1,70 @@ +import { IGDriveFiles } from "types/api/files"; +import { APIFilesResponse } from "types/api/response"; + +const mockFiles: APIFilesResponse = { + timestamp: Date.now(), + responseTime: 0, + folders: [], + files: [], + nextPageToken: "nextpageToken", + readmeExists: true, + bannerExists: true, + passwordExists: false, +}; +const mockFilesProtected: APIFilesResponse = { + timestamp: Date.now(), + responseTime: 0, + folders: [], + files: [], + nextPageToken: "nextpageToken", + readmeExists: true, + bannerExists: true, + passwordExists: true, +}; + +for (let i = 0; i < 10; i++) { + const folder: IGDriveFiles = { + mimeType: "application/vnd.google-apps.folder", + encryptedId: `encryptedId${i + 1}`, + name: `folder${i + 1}`, + trashed: false, + modifiedTime: new Date().toISOString(), + }; + const file: IGDriveFiles = { + mimeType: "application/vnd.google-apps.file", + encryptedId: `encryptedId${i + 1}`, + name: `file${i + 1}`, + trashed: false, + modifiedTime: new Date().toISOString(), + fileExtension: "txt", + encryptedWebContentLink: "encryptedWebContentLink", + size: Math.floor(Math.random() * 1000000000), + thumbnailLink: "/og.png", + imageMediaMetadata: + Math.random() > 0.5 + ? { + width: 1920, + height: 1080, + rotation: 0, + } + : null, + videoMediaMetadata: + Math.random() > 0.5 + ? { + width: 1920, + height: 1080, + durationMillis: 100000, + } + : null, + }; + + mockFiles.folders.push(folder); + mockFiles.files.push(file); + + folder.name = `folder${i + 1}protected`; + file.name = `file${i + 1}protected`; + mockFilesProtected.folders.push(folder); + mockFilesProtected.files.push(file); +} + +export { mockFiles, mockFilesProtected }; diff --git a/next.config.js b/next.config.js index a843cbe..8851780 100644 --- a/next.config.js +++ b/next.config.js @@ -1,6 +1,34 @@ /** @type {import('next').NextConfig} */ -const nextConfig = { - reactStrictMode: true, -} -module.exports = nextConfig +const withMDX = require("@next/mdx")({ + extension: /\.mdx?$/, + options: { + // If you use remark-gfm, you'll need to use next.config.mjs + // as the package is ESM only + // https://github.com/remarkjs/remark-gfm#install + remarkPlugins: [], + rehypePlugins: [], + // If you use `MDXProvider`, uncomment the following line. + providerImportSource: "@mdx-js/react", + }, +}); + +const nextConfig = { + pageExtensions: ["ts", "tsx", "js", "jsx", "md", "mdx"], + reactStrictMode: true, + async headers() { + return [ + { + source: "/:path*", + headers: [ + { + key: "Cache-Control", + value: "max-age=300, s-maxage=300, stale-while-revalidate, public", + }, + ], + }, + ]; + }, +}; + +module.exports = withMDX(nextConfig); diff --git a/package.json b/package.json index 2fee51f..5e77880 100644 --- a/package.json +++ b/package.json @@ -3,62 +3,62 @@ "version": "0.1.0", "private": true, "scripts": { - "dev": "next dev -p 5000", + "dev": "next dev -p 3000", "build": "next build", "start": "next start", "lint": "next lint", - "cypress": "cypress open" + "cypress": "cypress open", + "pretty": "prettier --write ." }, "dependencies": { - "@react-three/drei": "^9.66.1", - "@react-three/fiber": "^8.13.0", - "@types/react": "18.0.35", - "@types/react-dom": "18.0.11", - "@upstash/redis": "^1.20.4", - "@vercel/og": "^0.5.4", - "autoprefixer": "10.4.14", + "@cyntler/react-doc-viewer": "^1.13.0", + "@iconify/react": "^4.1.1", + "@mdx-js/loader": "^2.3.0", + "@mdx-js/react": "^2.3.0", + "@next/mdx": "^13.4.19", + "@popperjs/core": "^2.11.8", "axios": "^1.3.5", - "cors": "^2.8.5", "googleapis": "^118.0.0", "jsonwebtoken": "^9.0.0", - "mime-types": "^2.1.35", - "next": "13.3.0", - "next-seo": "^6.0.0", - "nextjs-progressbar": "^0.0.16", - "postcss": "8.4.22", - "react": "18.2.0", - "react-dom": "18.2.0", + "jszip": "^3.10.1", + "next": "^13.4.13", + "next-seo": "^6.1.0", + "react": "^18.2.0", + "react-dom": "^18.2.0", "react-h5-audio-player": "^3.8.6", - "react-icons": "^4.8.0", - "react-loading": "^2.0.3", + "react-is": "^18.2.0", "react-markdown": "^8.0.7", - "react-player": "^2.12.0", - "react-toastify": "^9.1.2", - "rehype-katex": "^6.0.2", - "rehype-prism-plus": "^1.5.1", - "rehype-raw": "^6.1.1", - "rehype-sanitize": "^5.0.1", - "remark-breaks": "^3.0.2", + "react-player": "^2.13.0", + "react-popper": "^2.3.0", + "react-toastify": "^9.1.3", + "rehype-katex": "^6.0.3", + "rehype-prism-plus": "^1.6.3", + "rehype-raw": "6.1.1", "remark-gfm": "^3.0.1", "remark-math": "^5.1.1", "remark-slug": "^7.0.1", - "remark-toc": "^8.0.1", - "swr": "^2.1.3", - "tailwindcss": "3.3.1", - "three": "^0.152.0" + "remark-toc": "^8.0.1" }, "devDependencies": { + "@faker-js/faker": "^8.0.2", "@tailwindcss/typography": "^0.5.9", + "@trivago/prettier-plugin-sort-imports": "^4.1.1", "@types/cors": "^2.8.13", "@types/jsonwebtoken": "^9.0.1", "@types/mime-types": "^2.1.1", "@types/node": "18.15.11", + "@types/react": "18.0.35", + "@types/react-dom": "18.0.11", "@types/three": "^0.150.2", - "cypress": "^12.10.0", + "autoprefixer": "10.4.14", + "encoding": "^0.1.13", "eslint": "8.38.0", - "eslint-config-next": "13.3.0", + "eslint-config-next": "^13.4.3", + "postcss": "8.4.22", "prettier": "^2.8.7", "prettier-plugin-tailwindcss": "^0.2.7", + "tailwind-merge": "^1.14.0", + "tailwindcss": "3.3.1", "typescript": "5.0.4" } } diff --git a/public/flaticon.svg b/public/flaticon.svg new file mode 100644 index 0000000..aed7d74 --- /dev/null +++ b/public/flaticon.svg @@ -0,0 +1,40 @@ + + + diff --git a/public/images/404.png b/public/images/404.png new file mode 100644 index 0000000..0cf89fa Binary files /dev/null and b/public/images/404.png differ diff --git a/public/images/password.png b/public/images/password.png new file mode 100644 index 0000000..0ee8346 Binary files /dev/null and b/public/images/password.png differ diff --git a/public/setup/FinalStep.md b/public/setup/FinalStep.md deleted file mode 100644 index 405dcec..0000000 --- a/public/setup/FinalStep.md +++ /dev/null @@ -1,26 +0,0 @@ -To finishing the setup, you need to do the following steps: - -## API Config -The API Config file are located at `src/config/api.ts`. You need to fill in the following information: -```js -module.exports = { - client_id: "lorem", - client_secret: "ipsum", - refresh_token: "dolor", -} -``` - -## Environment Config -The environment variables are differ for each hosting service. If you are using Vercel, you can add the environment variables on the project settings page. -```dotenv -ENCRYPTION_KEY="Encryption" -``` - -**NOTE:** Make sure to redeploy the site after adding the environment variables. - -## Customizing the Site -The site can be customized by editing the `src/config/site.ts` file. You can change the site title, description, and other information. -All the explanation also included in the file. - - -And that's it, you're done with the setup. You can now start using the site. \ No newline at end of file diff --git a/public/setup/GoogleCloudStep.md b/public/setup/GoogleCloudStep.md deleted file mode 100644 index 5425a23..0000000 --- a/public/setup/GoogleCloudStep.md +++ /dev/null @@ -1,47 +0,0 @@ -# Google Cloud Setup - -Table of Contents: -- [Making a Google Cloud Project and enabling Google Drive API](#making-a-google-cloud-project-and-enabling-google-drive-api) -- [Creating Oauth Consent Screen](#creating-oauth-consent-screen) -- [Creating Oauth Client ID](#creating-oauth-client-id) -- [Getting Refresh Token](#getting-refresh-token) - -## Making a Google Cloud Project and enabling Google Drive API -1. Open [Google Cloud Console](https://console.cloud.google.com/). -2. Click on the project selector dropdown in the top left corner and select or create a new project. - -3. Activate the [Google Drive API](https://console.cloud.google.com/apis/library/drive.googleapis.com) for your project. - -4. After activating the API, you will be redirected to the API overview page. For creating Oauth Client ID, you need to create Oauth consent first, you can do this by clicking on the "Oauth Consent Screen" menu. - -## Creating Oauth Consent Screen -1. On the Oauth consent screen page, select "External" as the user type and click on the "Create" button. - -2. On the next page, fill in the required information and click on the "Save" button. -(In my case I'm only filling `App name`, `User support email`, and `Developer contact information`.) -3. On the **Scopes** Page, select the following scopes and click on the "Save" button, then click on the "Continue" button. - -4. On the **Test Users** Page, click on the "Add Users" button and add your email address or any other email address that you want to use. Then click on the "Save and Continue" button. -5. And you're done with the Oauth consent screen. Now you can create the Oauth Client ID by clicking on the "Credentials" menu. -**NOTE: We don't need to publish the app, since we're only using it for our own purpose.** - -## Creating Oauth Client ID -1. On the **Credentials** page, click on the "Create Credentials" button and select "Oauth Client ID". - -2. On the next page, select "Web Application" as the application type, and fill in the required information. -3. In the **Authorized redirect URIs** section, add the following redirect URI: `https://developers.google.com/oauthplayground`, then click on the "Create" button. -4. After creating the Oauth Client ID, you will be redirected to the **Credentials** page. Then copy the Client ID, and Client Secret. (You will need them later on the bottom of this page, and to fetch refresh token.) - -## Getting Refresh Token -1. Open [Google OAuth Playground](https://developers.google.com/oauthplayground/). -2. Click on the gear icon on the upper right corner and check the box next to "Use your own OAuth credentials" and paste the Client ID and Client Secret that you copied earlier. - -3. Select the following scopes and click on the "Authorize APIs" button. - -4. Google will ask you to log in to your Google account, you need to log in using the account that specified on Test User (Creating Oauth Consent Screen - step 4). -5. It will show you a warning that the app is not verified, simply click on "Continue" button, and it will redirect you back to the Playground. - -6. Click on the "Exchange authorization code for tokens" button, and it will show you the refresh token. Copy the refresh token and paste it on the bottom of this page. - -7. You can close the Playground tab now. -8. You're done with the Google Cloud Setup. \ No newline at end of file diff --git a/public/setup/googleCloud.md b/public/setup/googleCloud.md new file mode 100644 index 0000000..e69de29 diff --git a/src/app/api/download/[encryptedId]/[fileName]/route.ts b/src/app/api/download/[encryptedId]/[fileName]/route.ts new file mode 100644 index 0000000..1837237 --- /dev/null +++ b/src/app/api/download/[encryptedId]/[fileName]/route.ts @@ -0,0 +1,94 @@ +import gIndexConfig from "config"; +import { NextRequest, NextResponse } from "next/server"; + +import getSearchParams from "utils/apiHelper/getSearchParams"; +import { decryptData } from "utils/encryptionHelper/hash"; +import ExtendedError from "utils/extendedError"; +import gdrive from "utils/gdriveInstance"; +import { isDownloadTokenValid } from "utils/tokenHelper"; + +import { ErrorResponse } from "types/api/response"; + +export async function GET( + request: NextRequest, + { + params, + }: { + params: { + encryptedId: string; + fileName: string; + }; + }, +) { + const reqStart = Date.now(); + const { encryptedId, fileName } = params; + try { + const data = JSON.parse(decryptData(encryptedId)) as { + id: string; + isProtected: boolean; + }; + const { token, preview, disableLimit } = getSearchParams(request.url, ["token", "preview", "disableLimit"]); + + if (!gIndexConfig.apiConfig.allowDownloadProtectedFile && data.isProtected && !token) + throw new ExtendedError("Missing download token", 403, "You are not authorized to download this file"); + if (!gIndexConfig.apiConfig.allowDownloadProtectedFile && data.isProtected && token && !isDownloadTokenValid(token)) + throw new ExtendedError("Invalid download token", 403, "You are not authorized to download this file"); + + const _fileMeta = gdrive.files.get({ + fileId: decryptData(data.id), + fields: "id, name, size, mimeType, webContentLink", + }); + const _fileContent = gdrive.files.get( + { + fileId: decryptData(data.id), + alt: "media", + }, + { responseType: "stream" }, + ); + + const [fileMeta, fileContent] = await Promise.all([_fileMeta, _fileContent]); + const fileSize = Number(fileMeta.data.size) ?? 0; + if (fileSize > gIndexConfig.apiConfig.maxFileSize && !disableLimit) { + return NextResponse.redirect(fileMeta.data.webContentLink!, { + status: 302, + headers: { + "Cache-Control": gIndexConfig.cacheControl, + }, + }); + } + const fileBuffer = await new Promise{errorMessage}
-
-
- {message || "Internal server error"}
-
-
- {message}
+
+
+ {props.path}
+
+ Please enter the password to continue:
+ {item.value ?? "-"}
+
+ {props.children}
+
+
+ {children}
+
+ ),
+ }}
+ >
+ {props.content}
+ + Preview only shows the first 5 pages. Please download the file to view the rest. ++ {isLoading ? ( +
- The folder or file you are trying to access is protected with a
- password.
-
- Please enter the password to continue.
-
+
-
-