diff --git a/GeoSAM-Image-Encoder/examples/data/setting.json b/GeoSAM-Image-Encoder/examples/data/setting.json deleted file mode 100644 index 6190407..0000000 --- a/GeoSAM-Image-Encoder/examples/data/setting.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "inputs": { - "INPUT": "/content/beiluhe_google_img_201211_clip.tif", - "BANDS": [ - 1, - 1, - 1 - ], - "RANGE": "0.0,255.0", - "CRS": "EPSG:32646", - "EXTENT": "471407.9709, 473331.8546, 3882162.2353, 3884389.1008 [EPSG:32646]", - "RESOLUTION": 0.9999395530145561, - "STRIDE": 512, - "CKPT": "/content/sam_vit_l_0b3195.pth", - "MODEL_TYPE": 1, - "BATCH_SIZE": 1, - "CUDA_ID": 0 - } -} \ No newline at end of file diff --git a/GeoSAM-Image-Encoder/examples/geosam-image-encoder.ipynb b/GeoSAM-Image-Encoder/examples/geosam-image-encoder.ipynb deleted file mode 100644 index b74859d..0000000 --- a/GeoSAM-Image-Encoder/examples/geosam-image-encoder.ipynb +++ /dev/null @@ -1,632 +0,0 @@ -{ - "nbformat": 4, - "nbformat_minor": 0, - "metadata": { - "colab": { - "provenance": [], - "gpuType": "T4", - "mount_file_id": "1nlvcmIyizdxZuJ66IapWW-RX1PlN4stW", - "authorship_tag": "ABX9TyOE/BHqrQ8wu+k+s9zXeIEz", - "include_colab_link": true - }, - "kernelspec": { - "name": "python3", - "display_name": "Python 3" - }, - "language_info": { - "name": "python" - }, - "accelerator": "GPU" - }, - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "view-in-github", - "colab_type": "text" - }, - "source": [ - "\"Open" - ] - }, - { - "cell_type": "markdown", - "source": [ - "# GeoSAM-Image-Encoder\n", - "\n", - "This package is part of Geo-SAM and is used to encode image data into features recognized by Geo-SAM." - ], - "metadata": { - "id": "ofyKNeqP6Wj4" - } - }, - { - "cell_type": "markdown", - "source": [ - "## Installation\n", - "\n", - "Installing `GeoSAM-Image-Encoder` directly will install the CPU version of `PyTorch`. Therefore, it is recommended to install the appropriate version of `PyTorch` before installing `GeoSAM-Image-Encoder` in your machine. You can install the corresponding version based on the official PyTorch website:\n", - "\n", - "\n", - "After installing PyTorch, you can install `GeoSAM-Image-Encoder` via pip.\n" - ], - "metadata": { - "id": "yd7SYS1o6NTo" - } - }, - { - "cell_type": "markdown", - "source": [ - "In Colab, PyTorch is already built-in, so you can install it directly." - ], - "metadata": { - "id": "dM0YGfCtsfUI" - } - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "x_LOrmGt3Gkh" - }, - "outputs": [], - "source": [ - "!pip install GeoSAM-Image-Encoder\n", - "# or\n", - "# !pip install git+https://github.com/Fanchengyan/GeoSAM-Image-Encoder.git" - ] - }, - { - "cell_type": "markdown", - "source": [ - "Download example dataset and sam `vit_l` checkpoint" - ], - "metadata": { - "id": "dM9ztEAm62Kl" - } - }, - { - "cell_type": "code", - "source": [ - "!wget https://raw.githubusercontent.com/coolzhao/Geo-SAM/main/rasters/beiluhe_google_img_201211_clip.tif\n", - "!wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_l_0b3195.pth\n", - "!wget https://raw.githubusercontent.com/coolzhao/Geo-SAM/dev/GeoSAM-Image-Encoder/examples/data/setting.json" - ], - "metadata": { - "id": "v32Lb6YW5FNU" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "source": [ - "## Usage\n", - "\n", - "You can call this script in Python or Terminal. We recommend using Python interface directly for processing, which will have greater flexibility." - ], - "metadata": { - "id": "ILKiN60dXhQF" - } - }, - { - "cell_type": "markdown", - "source": [ - "### Using Python\n", - "\n", - "After install GeoSAM-Image-Encoder, you can import it using `geosam`" - ], - "metadata": { - "id": "96Dof82l31rk" - } - }, - { - "cell_type": "code", - "source": [ - "import geosam\n", - "from geosam import ImageEncoder" - ], - "metadata": { - "id": "Z0K8RQV63H_v" - }, - "execution_count": 3, - "outputs": [] - }, - { - "cell_type": "markdown", - "source": [ - "check the folder contains geosam and add it to environment if you want to run in terminal" - ], - "metadata": { - "id": "34LNsf9H8oIA" - } - }, - { - "cell_type": "code", - "source": [ - "geosam.folder" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 35 - }, - "id": "LKQcboGt7US4", - "outputId": "6973806f-77b6-46b5-8a99-8babfe14aab2" - }, - "execution_count": null, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "'/usr/local/lib/python3.10/dist-packages/geosam'" - ], - "application/vnd.google.colaboratory.intrinsic+json": { - "type": "string" - } - }, - "metadata": {}, - "execution_count": 4 - } - ] - }, - { - "cell_type": "code", - "source": [ - "# check if gpu available\n", - "geosam.gpu_available()" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "VwvXGZLYS5LZ", - "outputId": "9647f83c-7416-4dae-dd2a-5ca0a762b979" - }, - "execution_count": null, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "True" - ] - }, - "metadata": {}, - "execution_count": 5 - } - ] - }, - { - "cell_type": "markdown", - "source": [ - "#### Run by specify parameters directly" - ], - "metadata": { - "id": "gb03VNJe4O2r" - } - }, - { - "cell_type": "code", - "source": [ - "checkpoint_path = '/content/sam_vit_l_0b3195.pth'\n", - "image_path = '/content/beiluhe_google_img_201211_clip.tif'\n", - "feature_dir = './'\n", - "\n", - "## init ImageEncoder\n", - "img_encoder = ImageEncoder(checkpoint_path)\n", - "## encode image\n", - "img_encoder.encode_image(image_path,feature_dir)" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "mNwD3v3D8RD1", - "outputId": "d627c917-1736-442c-c698-61199e762c58" - }, - "execution_count": 4, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "Initializing SAM model...\n", - "\n", - "\n", - "----------------------------------------------\n", - " Start encoding image to SAM features\n", - "----------------------------------------------\n", - "\n", - "Input Parameters:\n", - "----------------------------------------------\n", - " Input data value range to be rescaled: [0, 255] (automatically set based on min-max value of input image inside the processing extent.)\n", - " Image path: /content/beiluhe_google_img_201211_clip.tif\n", - " Bands selected: ['1', '2', '3']\n", - " Target resolution: 0.9999395530145561\n", - " Processing extent: [471407.9709, 3882162.2353, 473331.8546, 3884389.1008]\n", - " Processing image size: (width 1924, height 2227)\n", - "----------------------------------------------\n", - "\n", - "\n", - "RasterDataset info \n", - "----------------------------------------------\n", - " filename_glob: beiluhe_google_img_201211_clip.tif, \n", - " all bands: ['1', '2', '3', '4'], \n", - " input bands: ['1', '2', '3'], \n", - " resolution: 0.9999395530145561, \n", - " bounds: [471407.9709, 473331.8546571067, 3882162.2353493366, 3884389.1008, 0.0, 9.223372036854776e+18], \n", - " num: 1\n", - "----------------------------------------------\n", - "\n", - "----------------------------------------------\n", - " SAM model initialized. \n", - " SAM model type: vit_l\n", - " Device type: cuda:0\n", - " Patch size: (1024, 1024) \n", - " Batch size: 1\n", - " Patch sample num: 12\n", - " Total batch num: 12\n", - "----------------------------------------------\n", - "\n" - ] - }, - { - "output_type": "stream", - "name": "stderr", - "text": [ - "Encoding image: 100%|██████████| 12/12 [00:20<00:00, 1.70s/batch]" - ] - }, - { - "output_type": "stream", - "name": "stdout", - "text": [ - "\"Output feature path\": .\n" - ] - }, - { - "output_type": "stream", - "name": "stderr", - "text": [ - "\n" - ] - } - ] - }, - { - "cell_type": "markdown", - "source": [ - "#### Run by parameters from setting.json file\n", - "\n", - "If you want to using settings.json file to provide the parameters" - ], - "metadata": { - "id": "SuHYf5BQTT1H" - } - }, - { - "cell_type": "code", - "source": [ - "setting_file = \"/content/setting.json\"\n", - "feature_dir = './'\n", - "\n", - "### parse settings from the setting,json file\n", - "settings = geosam.parse_settings_file(setting_file)\n", - "\n", - "### setting file not contains feature_dir, you need add it\n", - "settings.update({\"feature_dir\":feature_dir})\n", - "\n", - "### split settings into init_settings, encode_settings\n", - "init_settings, encode_settings = geosam.split_settings(settings)\n", - "\n", - "print(f\"settings: {settings}\")\n", - "print(f\"init_settings: {init_settings}\")\n", - "print(f\"encode_settings: {encode_settings}\")" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "4r3xTViM91YL", - "outputId": "7b9fefbf-f94c-45b0-d58f-8380be69a1b6" - }, - "execution_count": null, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "settings: {'image_path': '/content/beiluhe_google_img_201211_clip.tif', 'bands': [1, 1, 1], 'value_range': '0.0,255.0', 'extent': '471407.9709, 473331.8546, 3882162.2353, 3884389.1008 [EPSG:32646]', 'resolution': 0.9999395530145561, 'stride': 512, 'checkpoint_path': '/content/sam_vit_l_0b3195.pth', 'model_type': 1, 'batch_size': 1, 'gpu_id': 0, 'feature_dir': './'}\n", - "init_settings: {'checkpoint_path': '/content/sam_vit_l_0b3195.pth', 'model_type': 1, 'batch_size': 1, 'gpu_id': 0}\n", - "encode_settings: {'image_path': '/content/beiluhe_google_img_201211_clip.tif', 'bands': [1, 1, 1], 'value_range': '0.0,255.0', 'extent': '471407.9709, 473331.8546, 3882162.2353, 3884389.1008 [EPSG:32646]', 'resolution': 0.9999395530145561, 'stride': 512, 'feature_dir': './'}\n" - ] - } - ] - }, - { - "cell_type": "code", - "source": [ - "## Then, you can run image encoding by parameters from setting.json file\n", - "img_encoder = ImageEncoder(**init_settings)\n", - "img_encoder.encode_image(**encode_settings)" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "y7fMTVTtFyjb", - "outputId": "5fc4cd7b-401a-4197-8d3f-e973de1b855e" - }, - "execution_count": null, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "Initializing SAM model...\n", - "\n", - "\n", - "----------------------------------------------\n", - " Start encoding image to SAM features\n", - "----------------------------------------------\n", - "\n", - "Input Parameters:\n", - "----------------------------------------------\n", - " Input data value range to be rescaled: (0.0, 255.0) (set by user)\n", - " Image path: /content/beiluhe_google_img_201211_clip.tif\n", - " Bands selected: ['1', '1', '1']\n", - " Target resolution: 0.9999395530145561\n", - " Processing extent: (471407.9709, 473331.8546, 3882162.2353, 3884389.1008)\n", - " Processing image size: (width 3410960, height 3411263)\n", - "----------------------------------------------\n", - "\n", - "\n", - "RasterDataset info \n", - "----------------------------------------------\n", - " filename_glob: beiluhe_google_img_201211_clip.tif, \n", - " all bands: ['1', '2', '3', '4'], \n", - " input bands: ['1', '1', '1'], \n", - " resolution: 0.9999395530145561, \n", - " bounds: [471407.9709, 473331.8546571067, 3882162.2353493366, 3884389.1008, 0.0, 9.223372036854776e+18], \n", - " num: 1\n", - "----------------------------------------------\n", - "\n", - "----------------------------------------------\n", - " SAM model initialized. \n", - " SAM model type: vit_l\n", - " Device type: cuda:0\n", - " Patch size: (1024, 1024) \n", - " Batch size: 1\n", - " Patch sample num: 12\n", - " Total batch num: 12\n", - "----------------------------------------------\n", - "\n" - ] - }, - { - "output_type": "stream", - "name": "stderr", - "text": [ - "Encoding image: 100%|██████████| 12/12 [00:13<00:00, 1.11s/batch]" - ] - }, - { - "output_type": "stream", - "name": "stdout", - "text": [ - "\"Output feature path\": .\n" - ] - }, - { - "output_type": "stream", - "name": "stderr", - "text": [ - "\n" - ] - } - ] - }, - { - "cell_type": "markdown", - "source": [ - "### Using Terminal\n", - "\n", - "Since this is a Colab example, Python will be used to demonstrate running it in the terminal." - ], - "metadata": { - "id": "0YiFNWuz4iWQ" - } - }, - { - "cell_type": "code", - "source": [ - "import os\n", - "\n", - "## change cwd to geosam folder\n", - "os.chdir(geosam.folder)\n", - "print(os.getcwd())" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "YDr4oyCKHMpV", - "outputId": "7c86845d-006b-4be5-b2f1-b670e9510d44" - }, - "execution_count": null, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "/usr/local/lib/python3.10/dist-packages/geosam\n" - ] - } - ] - }, - { - "cell_type": "code", - "source": [ - "## get the command for terminal\n", - "cmd = f\"image_encoder.py -i {image_path} -c {checkpoint_path} -f {feature_dir}\"\n", - "print(cmd)" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "vD686kzAXR6G", - "outputId": "d6efe171-366a-4d4a-c10a-490f59312d5b" - }, - "execution_count": null, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "image_encoder.py -i /content/beiluhe_google_img_201211_clip.tif -c /content/sam_vit_l_0b3195.pth -f ./\n" - ] - } - ] - }, - { - "cell_type": "code", - "source": [ - "## run in terminal\n", - "!python image_encoder.py -i /content/beiluhe_google_img_201211_clip.tif -c /content/sam_vit_l_0b3195.pth -f ./\n", - "\n", - "## You can overwrite the settings from file by specify the parameter values. For Example:\n", - "# !python image_encoder.py -s /content/setting.json -f ./ --stride 256 --value_range \"10,255\"" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "W_LbTprzXa0M", - "outputId": "7d264629-c1a5-4c89-feec-b11736831198" - }, - "execution_count": null, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "\n", - "settings:\n", - " {'feature_dir': PosixPath('/usr/local/lib/python3.10/dist-packages/geosam'), 'image_path': PosixPath('/content/beiluhe_google_img_201211_clip.tif'), 'checkpoint_path': PosixPath('/content/sam_vit_l_0b3195.pth'), 'stride': 512, 'batch_size': 1, 'gpu_id': 0}\n", - "\n", - "Initializing SAM model...\n", - "\n", - "\n", - "----------------------------------------------\n", - " Start encoding image to SAM features\n", - "----------------------------------------------\n", - "\n", - "Input Parameters:\n", - "----------------------------------------------\n", - " Input data value range to be rescaled: [0, 255] (automatically set based on min-max value of input image inside the processing extent.)\n", - " Image path: /content/beiluhe_google_img_201211_clip.tif\n", - " Bands selected: ['1', '2', '3']\n", - " Target resolution: 0.9999395530145561\n", - " Processing extent: [471407.9709, 3882162.2353, 473331.8546, 3884389.1008]\n", - " Processing image size: (width 1924, height 2227)\n", - "----------------------------------------------\n", - "\n", - "\n", - "RasterDataset info \n", - "----------------------------------------------\n", - " filename_glob: beiluhe_google_img_201211_clip.tif, \n", - " all bands: ['1', '2', '3', '4'], \n", - " input bands: ['1', '2', '3'], \n", - " resolution: 0.9999395530145561, \n", - " bounds: [471407.9709, 473331.8546571067, 3882162.2353493366, 3884389.1008, 0.0, 9.223372036854776e+18], \n", - " num: 1\n", - "----------------------------------------------\n", - "\n", - "----------------------------------------------\n", - " SAM model initialized. \n", - " SAM model type: vit_l\n", - " Device type: cuda:0\n", - " Patch size: (1024, 1024) \n", - " Batch size: 1\n", - " Patch sample num: 12\n", - " Total batch num: 12\n", - "----------------------------------------------\n", - "\n", - "Encoding image: 100% 12/12 [00:13<00:00, 1.16s/batch]\n", - "\"Output feature path\": /usr/local/lib/python3.10/dist-packages/geosam\n" - ] - } - ] - }, - { - "cell_type": "code", - "source": [ - "## check all available parameters:\n", - "!python image_encoder.py -h" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "4aXYVNWVyUbg", - "outputId": "5da59b2f-dc24-46fa-f6aa-b0bc6ae13097" - }, - "execution_count": null, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "\n", - "This script is for encoding image to SAM features.\n", - "\n", - "=====\n", - "Usage\n", - "=====\n", - "using settings.json:\n", - "\n", - " image_encoder.py -s -f \n", - " \n", - " \n", - "or directly using parameters:\n", - " \n", - " image_encoder.py -i -c -f \n", - " \n", - "All Parameters:\n", - "-------------------\n", - "-s, --settings: Path to the settings json file.\n", - "-i, --image_path: Path to the input image.\n", - "-c, --checkpoint_path: Path to the SAM checkpoint.\n", - "-f, --feature_dir: Path to the output feature directory.\n", - "--model_type: one of [\"vit_h\", \"vit_l\", \"vit_b\"] or [0, 1, 2] or None, optional\n", - " The type of the SAM model. If None, the model type will be \n", - " inferred from the checkpoint path. Default: None. \n", - "--bands: list of int, optional .\n", - " The bands to be used for encoding. Should not be more than three bands.\n", - " If None, the first three bands (if available) will be used. Default: None.\n", - "--stride: int, optional\n", - " The stride of the sliding window. Default: 512.\n", - "--extent: str, optional\n", - " The extent of the image to be encoded. Should be in the format of\n", - " \"minx, miny, maxx, maxy, [crs]\". If None, the extent of the input\n", - " image will be used. Default: None.\n", - "--value_range: tuple of float, optional\n", - " The value range of the input image. If None, the value range will be\n", - " automatically calculated from the input image. Default: None.\n", - "--resolution: float, optional\n", - " The resolution of the output feature in the unit of raster crs.\n", - " If None, the resolution of the input image will be used. Default: None.\n", - "--batch_size: int, optional\n", - " The batch size for encoding. Default: 1.\n", - "--gpu_id: int, optional\n", - " The device id of the GPU to be used. Default: 0.\n", - "\n" - ] - } - ] - } - ] -} \ No newline at end of file diff --git a/GeoSAM-Image-Encoder/geosam/__init__.py b/GeoSAM-Image-Encoder/geosam/__init__.py deleted file mode 100644 index 8365588..0000000 --- a/GeoSAM-Image-Encoder/geosam/__init__.py +++ /dev/null @@ -1,4 +0,0 @@ -import os -from .image_encoder import * - -folder = os.path.dirname(os.path.abspath(__file__)) diff --git a/GeoSAM-Image-Encoder/geosam/image_encoder.py b/GeoSAM-Image-Encoder/geosam/image_encoder.py deleted file mode 100644 index 988fa15..0000000 --- a/GeoSAM-Image-Encoder/geosam/image_encoder.py +++ /dev/null @@ -1,654 +0,0 @@ -#!/usr/bin/env python -""" -This script is for encoding image to SAM features. - -===== -Usage -===== -using settings.json: - - image_encoder.py -s -f - - -or directly using parameters: - - image_encoder.py -i -c -f - -All Parameters: -------------------- --s, --settings: Path to the settings json file. --i, --image_path: Path to the input image. --c, --checkpoint_path: Path to the SAM checkpoint. --f, --feature_dir: Path to the output feature directory. ---model_type: one of ["vit_h", "vit_l", "vit_b"] or [0, 1, 2] or None, optional - The type of the SAM model. If None, the model type will be - inferred from the checkpoint path. Default: None. ---bands: list of int, optional . - The bands to be used for encoding. Should not be more than three bands. - If None, the first three bands (if available) will be used. Default: None. ---stride: int, optional - The stride of the sliding window. Default: 512. ---extent: str, optional - The extent of the image to be encoded. Should be in the format of - "minx, miny, maxx, maxy, [crs]". If None, the extent of the input - image will be used. Default: None. ---value_range: tuple of float, optional - The value range of the input image. If None, the value range will be - automatically calculated from the input image. Default: None. ---resolution: float, optional - The resolution of the output feature in the unit of raster crs. - If None, the resolution of the input image will be used. Default: None. ---batch_size: int, optional - The batch size for encoding. Default: 1. ---gpu_id: int, optional - The device id of the GPU to be used. Default: 0. -""" - -import sys -import getopt -import json -from pathlib import Path -from typing import List, Tuple, Union -from tqdm import tqdm -from segment_anything import sam_model_registry -from segment_anything.modeling import Sam -import torch -from geosam.torchgeo_sam import SamTestGridGeoSampler, SamTestRasterDataset -from torchgeo.samplers import Units -from torchgeo.datasets import BoundingBox, stack_samples -from torch.utils.data import DataLoader -import rasterio -from rasterio import warp -from rasterio.crs import CRS -import numpy as np -import pandas as pd -from torch import Tensor -import hashlib - - -SAM_Model_Types = ["vit_h", - "vit_l", - "vit_b"] - -Parameter_Mapping = { - "INPUT": "image_path", - "CKPT": "checkpoint_path", - "MODEL_TYPE": "model_type", - "BANDS": "bands", - "STRIDE": "stride", - "EXTENT": "extent", - "RANGE": "value_range", - "RESOLUTION": "resolution", - "BATCH_SIZE": "batch_size", - "CUDA_ID": "gpu_id" -} - -Init_Settings = [ - 'checkpoint_path', - 'model_type', - 'batch_size', - 'gpu_id' -] -Encode_Settings = [ - 'image_path', - 'feature_dir', - 'bands', - 'stride', - 'extent', - 'value_range', - 'resolution' -] - - -class ImageEncoder: - '''Encode image to SAM features.''' - - def __init__( - self, - checkpoint_path: Union[str, Path], - model_type: str = None, - batch_size: int = 1, - gpu: bool = True, - gpu_id: int = 0, - ): - '''Initialize the ImageEncoder. - - Parameters: - ---------- - checkpoint_path: str or Path - Path to the SAM checkpoint. - model_type: one of ["vit_h", "vit_l", "vit_b"] or [0, 1, 2] or None, optional - The type of the SAM model. If None, the model type will be - inferred from the checkpoint path. Default: None. - batch_size: int, optional - The batch size for encoding. Default: 1. - gpu: bool, optional - Whether to use GPU for encoding if available. Default: True. - gpu_id: int, optional - The device id of the GPU to be used. Default: 0. - ''' - self.checkpoint_path = Path(checkpoint_path) - - self.batch_size = batch_size - self.gpu = gpu - self.gpu_id = gpu_id - self.model_type = check_model_type(model_type, self.checkpoint_path) - self.device = detect_device(self.gpu, self.gpu_id) - self.sam_model = self.initialize_sam() - - def encode_image( - self, - image_path: Union[str, Path], - feature_dir: Union[str, Path], - bands: List[int] = None, - stride: int = 512, - extent: str = None, - value_range: Tuple[float, float] = None, - resolution: float = None, - ): - '''Encode image to SAM features. - - Parameters: - ---------- - image_path: str or Path - Path to the input image. - feature_dir: str or Path - Path to the output feature directory. - bands: list of int, optional - The bands to be used for encoding. Should not be more than three bands. - If None, the first three bands (if available) will be used. Default: None. - stride: int, optional - The stride of the sliding window. Default: 512. - extent: str, optional - The extent of the image to be encoded. Should be in the format of - "minx, miny, maxx, maxy, [crs]". If None, the extent of the input - image will be used. Default: None. - value_range: tuple of float, optional - The value range of the input image. If None, the value range will be - automatically calculated from the input image. Default: None. - resolution: float, optional - The resolution of the output feature in the unit of raster crs. - If None, the resolution of the input image will be used. Default: None. - ''' - image_path = Path(image_path) - feature_dir = Path(feature_dir) - - print('\n----------------------------------------------') - print(' Start encoding image to SAM features') - print('----------------------------------------------\n') - - # load image and check extent - with rasterio.open(image_path) as src: - arr = src.read() - meta = src.meta.copy() - - if extent is None: - extent = [i for i in src.bounds] - extent_crs = src.crs - else: - extent_crs = extent.split(' ')[-1].strip()[1:-1].strip() - extent = [float(i.strip(',').strip()) - for i in extent.split(' ')[:-1]] - if extent_crs != '': - extent_crs = CRS.from_user_input(extent_crs) - extent = warp.transform_bounds( - extent_crs, - src.crs, - *extent, - ) - - # check bands - if bands is None: - max_band = min(3, meta['count']) - bands = list(range(1, max_band+1)) - - if len(bands) > 3: - raise ValueError( - "SAM only supports no more than three bands," - f" but {len(bands)} bands are given." - ) - if max(bands) > meta['count']: - raise ValueError( - f"The band number of the input image is {meta['count']}. " - f"But the band index {max(bands)} is given." - ) - # ensure only three bands are used, less than three bands will be broadcasted to three bands - bands = [str(i) for i in (bands * 3)[0:3]] - - # check resolution - if resolution is None: - resolution = meta['transform'][0] - - # get pixel number in the extent - img_width_in_extent = round( - (extent[2] - extent[0])/resolution) - img_height_in_extent = round( - (extent[3] - extent[1])/resolution) - - # Print input parameters - print('Input Parameters:') - print('----------------------------------------------') - # check value range - if value_range is not None: - if isinstance(value_range, str): - try: - value_range = eval(value_range) - except: - raise ValueError( - f"Could not evaluate the value range. {value_range}" - "Please check the format of the value range." - ) - if value_range[0] >= value_range[1]: - raise ValueError( - "Data value range is wrongly set or the image is with constant values.") - print(' Input data value range to be rescaled: ' - f'{value_range} (set by user)') - else: - value_range = [np.nanmin(arr), - np.nanmax(arr)] - print( - f' Input data value range to be rescaled: {value_range} ' - '(automatically set based on min-max value of input image inside the processing extent.)' - ) - - print(f' Image path: {image_path}') - print(f' Bands selected: {bands}') - print(f' Target resolution: {resolution}') - print(f' Processing extent: {extent}') - print(f' Processing image size: (width {img_width_in_extent}, ' - f'height {img_height_in_extent})') - print('----------------------------------------------\n') - - img_dir = str(image_path.parent) - img_name = image_path.name - - SamTestRasterDataset.filename_glob = img_name - SamTestRasterDataset.all_bands = [ - str(i) for i in range(1, meta['count']+1) - ] - - sam_ds = SamTestRasterDataset( - root=img_dir, - crs=None, - res=resolution, - bands=bands, - cache=False - ) - - print( - f'\nRasterDataset info ' - '\n----------------------------------------------' - f'\n filename_glob: {sam_ds.filename_glob}, ' - f'\n all bands: {sam_ds.all_bands}, ' - f'\n input bands: {sam_ds.bands}, ' - f'\n resolution: {sam_ds.res}, ' - f'\n bounds: {sam_ds.index.bounds}, ' - f'\n num: {len(sam_ds.index)}' - '\n----------------------------------------------\n' - ) - - extent_bbox = BoundingBox( - minx=extent[0], - maxx=extent[2], - miny=extent[1], - maxy=extent[3], - mint=sam_ds.index.bounds[4], - maxt=sam_ds.index.bounds[5] - ) - - ds_sampler = SamTestGridGeoSampler( - sam_ds, - size=self.sam_model.image_encoder.img_size, - stride=stride, - roi=extent_bbox, - units=Units.PIXELS # Units.CRS or Units.PIXELS - ) - - if len(ds_sampler) == 0: - raise ValueError( - f'!!!No available patch sample inside the chosen extent!!!') - - ds_dataloader = DataLoader( - sam_ds, - batch_size=self.batch_size, - sampler=ds_sampler, - collate_fn=stack_samples - ) - - print('----------------------------------------------') - print(f' SAM model initialized. \n ' - f' SAM model type: {self.model_type}') - - if not self.gpu or not gpu_available(): - print(' !!!No GPU available, using CPU instead!!!') - self.batch_size = 1 - print(f' Device type: {self.device}') - - print(f' Patch size: {ds_sampler.patch_size} \n' - f' Batch size: {self.batch_size}') - print(f' Patch sample num: {len(ds_sampler)}') - print(f' Total batch num: {len(ds_dataloader)}') - print('----------------------------------------------\n') - - for patch_idx, batch in tqdm( - enumerate(ds_dataloader), - desc='Encoding image', - unit='batch', - total=len(ds_dataloader) - ): - - batch_input = rescale_img( - batch_input=batch['image'], value_range=value_range) - - features = get_sam_feature(batch_input, self.sam_model) - - # print(f'\nBatch no. {patch_idx+1} loaded') - # print(f'img_shape: ' + str(batch['img_shape'][0])) - # print('patch_size: ' + str(batch['image'].shape)) - # print(f'feature_shape: {features.shape}') - - # TODO: show gpu usage info - - save_sam_feature( - sam_ds, - feature_dir, - batch, - features, - extent, - patch_idx, - self.model_type - ) - - print(f'"Output feature path": {feature_dir}') - - def initialize_sam(self) -> Sam: - print("Initializing SAM model...\n") - sam_model = sam_model_registry[self.model_type]( - checkpoint=self.checkpoint_path) - sam_model.to(device=self.device) - return sam_model - - -def check_model_type(model_type, checkpoint_path): - checkpoint_path = Path(checkpoint_path) - model_type_errors_str = ( - f"model_type should be one of {SAM_Model_Types} or {range(len(SAM_Model_Types))}, " - f"but {model_type} is given." - ) - if isinstance(model_type, int): - if model_type not in range(len(SAM_Model_Types)): - raise ValueError(model_type_errors_str) - else: - model_type = SAM_Model_Types[model_type] - elif isinstance(model_type, str): - if model_type not in SAM_Model_Types: - raise ValueError(model_type_errors_str) - elif model_type is None: - # infer model type from checkpoint path - flag = False - for model_type in SAM_Model_Types: - if model_type in checkpoint_path.name: - flag = True - break - if not flag: - raise ValueError( - "Could not infer the model type from the checkpoint path. " - "Please specify the model type manually." - ) - else: - raise ValueError(model_type_errors_str) - return model_type - - -def gpu_available() -> bool: - '''Check if GPU is available.''' - return cuda_available() or mps_available() - - -def cuda_available() -> bool: - '''Check if CUDA (NVIDIA or ROCm) is available.''' - return torch.cuda.is_available() - - -def mps_available() -> bool: - '''Check if MPS (Mac) is available.''' - return torch.backends.mps.is_available() - - -def detect_device(gpu: bool, gpu_id: int) -> str: - '''Detect device for pytorch.''' - if not gpu: - return 'cpu' - elif cuda_available(): - return f'cuda:{gpu_id}' - elif mps_available(): - return f'mps:{gpu_id}' - else: - print('No GPU available, using CPU instead.') - return 'cpu' - - -def rescale_img(batch_input: Tensor, value_range: List[float]) -> Tensor: - 'rescale input image to [0,255]' - range_min = value_range[0] - range_max = value_range[1] - batch_output = (batch_input - range_min)*255/(range_max - range_min) - return batch_output - - -@torch.no_grad() -def get_sam_feature(batch_input: Tensor, sam_model) -> bool: - # TODO: if the input image are all zero(batch_input.any()), directly return features with all zero and give a message - # should know the shape of the feature in advance - batch_input = batch_input.to(device=sam_model.device) - batch_input = ((batch_input - sam_model.pixel_mean) / - sam_model.pixel_std) - features = sam_model.image_encoder(batch_input) - return features.cpu().numpy() - - -def save_sam_feature( - raster_ds: SamTestRasterDataset, - export_dir: Path, - data_batch: Tensor, - feature: np.ndarray, - extent: List[float], - patch_idx: int, - model_type: str = "vit_h" -) -> int: - # iterate over batch_size dimension - for idx in range(feature.shape[-4]): - band_num = feature.shape[-3] - height = feature.shape[-2] - width = feature.shape[-1] - bbox = data_batch['bbox'][idx] - rio_transform = rasterio.transform.from_bounds( - bbox.minx, bbox.miny, bbox.maxx, bbox.maxy, width, height) # west, south, east, north, width, height - filepath = Path(data_batch['path'][idx]) - bbox_list = [bbox.minx, bbox.miny, bbox.maxx, bbox.maxy] - bbox_str = '_'.join(map("{:.6f}".format, bbox_list)) - extent_str = '_'.join( - map("{:.6f}".format, extent)) + f"_res_{raster_ds.res:.6f}" - # Unicode-objects must be encoded before hashing with hashlib and - # because strings in Python 3 are Unicode by default (unlike Python 2), - # you'll need to encode the string using the .encode method. - bbox_hash = hashlib.sha256(bbox_str.encode("utf-8")).hexdigest() - extent_hash = hashlib.sha256( - extent_str.encode("utf-8")).hexdigest() - - bands_str = '_'.join([str(band) for band in raster_ds.band_indexes]) - export_dir_sub = (export_dir / filepath.stem / - f"sam_feat_{model_type}_bands_{bands_str}_{extent_hash[0:16]}") - export_dir_sub.mkdir(parents=True, exist_ok=True) - feature_tiff = (export_dir_sub / - f"sam_feat_{model_type}_{bbox_hash}.tif") - feature_csv = (export_dir_sub / f"{export_dir_sub.name}.csv") - with rasterio.open( - feature_tiff, - mode="w", - driver="GTiff", - height=height, width=width, - count=band_num, - dtype='float32', - crs=data_batch['crs'][idx], - transform=rio_transform - ) as feature_dataset: - # index start from 1, feature[idx, :, :, :] = feature[idx, ...], later is faster - feature_dataset.write(feature[idx, ...], range(1, band_num+1)) - # pr_mask_dataset.set_band_description(1, '') - tags = { - "img_shape": data_batch["img_shape"][idx], - "input_shape": data_batch["input_shape"][idx], - "model_type": model_type, - } - feature_dataset.update_tags(**tags) - feature_crs = feature_dataset.crs - - index_df = pd.DataFrame(columns=['minx', 'maxx', 'miny', 'maxy', 'mint', 'maxt', - 'filepath', - 'crs', 'res'], - index=[patch_idx]) - index_df['filepath'] = [feature_tiff.name] - index_df['minx'] = [bbox.minx] - index_df['maxx'] = [bbox.maxx] - index_df['miny'] = [bbox.miny] - index_df['maxy'] = [bbox.maxy] - index_df['mint'] = [bbox.mint] - index_df['maxt'] = [bbox.maxt] - index_df['crs'] = [str(feature_crs)] - index_df['res'] = [raster_ds.res] - index_df['model_type'] = [model_type] - # append data frame to CSV file, index=False - index_df.to_csv(feature_csv, mode='a', - header=not feature_csv.exists(), index=True) - - -class Usage(Exception): - """Usage context manager""" - - def __init__(self, msg): - self.msg = msg - - -def encode_image_from_cmd(argv=None): - if argv == None: - argv = sys.argv - - settings_path = None - image_path = None - checkpoint_path = None - feature_dir = None - model_type = None - bands = None - stride = 512 - extent = None - value_range = None - resolution = None - batch_size = 1 - gpu_id = 0 - - try: - opts, args = getopt.getopt(argv[1:], "hs:i:c:f:", [ - "help", - "settings=", - "image_path=", - "checkpoint_path=", - "feature_dir=", - "model_type=", - "bands=", - "stride=", - "extent=", - "value_range=", - "resolution=", - "batch_size=", - "gpu_id=" - ]) - except getopt.error as msg: - raise Usage(msg) - - for o, a in opts: - if o in ['-h', '--help']: - print(__doc__) - return 0 - elif o in ['-s', '--settings']: - settings_path = Path(a).absolute() - elif o in ['-i', '--image_path']: - image_path = Path(a).absolute() - elif o in ['-c', '--checkpoint_path']: - checkpoint_path = Path(a).absolute() - elif o in ['-f', '--feature_dir']: - feature_dir = Path(a).absolute() - elif o == '--model_type': - model_type = a - elif o == '--bands': - bands = eval(a) - elif o == '--stride': - stride = int(a) - elif o == '--extent': - extent = a - elif o == '--value_range': - value_range = a - elif o == '--resolution': - resolution = float(a) - elif o == '--batch_size': - batch_size = int(a) - elif o == '--gpu_id': - gpu_id = int(a) - - settings = {} - if feature_dir is not None: - settings['feature_dir'] = feature_dir - else: - raise ValueError('feature_dir is not specified.') - - if settings_path is not None: - settings.update(parse_settings_file(settings_path)) - - if image_path is not None: - settings.update({'image_path': image_path}) - if checkpoint_path is not None: - settings.update({'checkpoint_path': checkpoint_path}) - if model_type is not None: - if len(model_type) == 1: - model_type = int(model_type) - settings.update({'model_type': model_type}) - if bands is not None: - settings.update({'bands': bands}) - if stride is not None: - settings.update({'stride': stride}) - if extent is not None: - settings.update({'extent': extent}) - if value_range is not None: - settings.update({'value_range': value_range}) - if resolution is not None: - settings.update({'resolution': resolution}) - if batch_size is not None: - settings.update({'batch_size': batch_size}) - if gpu_id is not None: - settings.update({'gpu_id': gpu_id}) - - print(f"\nsettings:\n {settings}\n") - init_settings, encode_settings = split_settings(settings) - img_encoder = ImageEncoder(**init_settings) - img_encoder.encode_image(**encode_settings) - - -def parse_settings_file(settings_path): - with open(settings_path) as f: - settings = json.load(f) - settings = settings['inputs'] - settings = {Parameter_Mapping[k]: v for k, - v in settings.items() if k != 'CRS'} - return settings - - -def split_settings(settings): - init_settings = {k: v for k, v in settings.items() - if k in Init_Settings} - encode_settings = {k: v for k, v in settings.items() - if k in Encode_Settings} - return init_settings, encode_settings - - -if __name__ == "__main__": - sys.exit(encode_image_from_cmd()) diff --git a/GeoSAM-Image-Encoder/geosam/torchgeo_sam.py b/GeoSAM-Image-Encoder/geosam/torchgeo_sam.py deleted file mode 100644 index a2382ab..0000000 --- a/GeoSAM-Image-Encoder/geosam/torchgeo_sam.py +++ /dev/null @@ -1,640 +0,0 @@ -# Extension of torchgeo library by zyzhao -import glob -import os -import re -import sys -from typing import (Any, Callable, Dict, Iterable, Iterator, List, Optional, - Sequence, Tuple, Union, cast) - -import matplotlib.pyplot as plt -import numpy as np -import pandas as pd -import rasterio -import torch -from rasterio.crs import CRS -from rasterio.enums import Resampling -from rasterio.vrt import WarpedVRT -from rasterio.windows import from_bounds as window_from_bounds -from rtree.index import Index, Property -from torch import Tensor -from torch.nn import functional as F -from torchgeo.datasets import BoundingBox, GeoDataset, RasterDataset -from torchgeo.datasets.utils import disambiguate_timestamp -from torchgeo.samplers import GeoSampler, Units -from torchgeo.samplers.constants import Units -from torchgeo.samplers.utils import _to_tuple, tile_to_chips - - -class SamTestGridGeoSampler(GeoSampler): - """Samples elements in a grid-like fashion. - accept image smaller than desired patch_size - """ - - def __init__( - self, - dataset: GeoDataset, - size: Union[Tuple[float, float], float], - stride: Union[Tuple[float, float], float], - roi: Optional[BoundingBox] = None, - units: Units = Units.PIXELS, - ) -> None: - """Initialize a new Sampler instance. - - The ``size`` and ``stride`` arguments can either be: - - * a single ``float`` - in which case the same value is used for the height and - width dimension - * a ``tuple`` of two floats - in which case, the first *float* is used for the - height dimension, and the second *float* for the width dimension - - .. versionchanged:: 0.3 - Added ``units`` parameter, changed default to pixel units - - Args: - dataset: dataset to index from - size: dimensions of each :term:`patch` - stride: distance to skip between each patch - roi: region of interest to sample from (minx, maxx, miny, maxy, mint, maxt) - (defaults to the bounds of ``dataset.index``) - units: defines if ``size`` and ``stride`` are in pixel or CRS units - """ - super().__init__(dataset, roi) - self.size = _to_tuple(size) - self.patch_size = self.size - self.stride = _to_tuple(stride) - - if units == Units.PIXELS: - self.size = (self.size[0] * self.res, self.size[1] * self.res) - self.stride = (self.stride[0] * self.res, - self.stride[1] * self.res) - - self.hits = [] - self.hits_small = [] - for hit in self.index.intersection(tuple(self.roi), objects=True): - bounds = BoundingBox(*hit.bounds) - if ( - bounds.maxx - bounds.minx >= self.size[1] - # change 'and' to 'or' for handling strip images - or bounds.maxy - bounds.miny >= self.size[0] - ): - self.hits.append(hit) - else: - self.hits_small.append(hit) - - self.length = 0 - for hit in self.hits: - bounds = BoundingBox(*hit.bounds) - rows, cols = tile_to_chips(bounds, self.size, self.stride) - self.length += rows * cols - - for hit in self.hits_small: - bounds = BoundingBox(*hit.bounds) - self.length += 1 - - def __iter__(self) -> Iterator[Dict[str, Any]]: - """Return the index of a dataset. - - Returns: - bbox(minx, maxx, miny, maxy, mint, maxt) coordinates to index a dataset - & raster file path - """ - # For each tile... - for hit in self.hits + self.hits_small: - if hit in self.hits: - bounds = BoundingBox(*hit.bounds) - rows, cols = tile_to_chips(bounds, self.size, self.stride) - mint = bounds.mint - maxt = bounds.maxt - - # For each row... - for i in range(rows): - miny = bounds.miny + i * self.stride[0] - maxy = miny + self.size[0] - if maxy > bounds.maxy: - maxy = bounds.maxy - miny = bounds.maxy - self.size[0] - if miny < bounds.miny: - miny = bounds.miny - - # For each column... - for j in range(cols): - minx = bounds.minx + j * self.stride[1] - maxx = minx + self.size[1] - if maxx > bounds.maxx: - maxx = bounds.maxx - minx = bounds.maxx - self.size[1] - if minx < bounds.minx: - minx = bounds.minx - - query = { - "bbox": BoundingBox(minx, maxx, miny, maxy, mint, maxt), - "path": cast(str, hit.object), - "size": int(self.patch_size[0]) - } - - # BoundingBox(minx, maxx, miny, maxy, mint, maxt) - yield query - else: - bounds = BoundingBox(*hit.bounds) - minx = bounds.minx - miny = bounds.miny - maxx = bounds.maxx - maxy = bounds.maxy - mint = bounds.mint - maxt = bounds.maxt - query = { - "bbox": BoundingBox(minx, maxx, miny, maxy, mint, maxt), - "path": cast(str, hit.object), - "size": int(self.patch_size[0]) - } - - yield query - - def __len__(self) -> int: - """Return the number of samples over the ROI. - - Returns: - number of patches that will be sampled - """ - return self.length - - -class SamTestFeatureDataset(RasterDataset): - filename_glob = "*.tif" - # filename_regex = r"^S2.{5}_(?P\d{8})_N\d{4}_R\d{3}_6Bands_S\d{1}" - filename_regex = ".*" - date_format = "" - is_image = True - separate_files = False - all_bands = [] - rgb_bands = [] - - def __init__(self, root: str = "data", - crs: Optional[CRS] = None, - res: Optional[float] = None, - bands: Optional[Sequence[str]] = None, - transforms: Optional[Callable[[ - Dict[str, Any]], Dict[str, Any]]] = None, - cache: bool = True - ) -> None: - if self.separate_files: - raise NotImplementedError( - 'Testing for separated files are not supported yet' - ) - # super().__init__(root, crs, res, bands, transforms, cache) - self.transforms = transforms - - # Create an R-tree to index the dataset - self.index = Index(interleaved=False, properties=Property(dimension=3)) - - self.root = root - self.bands = bands or self.all_bands - self.cache = cache - self.model_type = None - model_type_list = ["vit_h", "vit_l", "vit_b"] - - # Populate the dataset index - i = 0 - # pathname = os.path.join(root, "**", self.filename_glob) - pathname = os.path.join(root, self.filename_glob) - raster_list = glob.glob(pathname, recursive=True) - raster_name = os.path.basename(raster_list[0]) - for m_type in model_type_list: - if m_type in raster_name: - self.model_type = m_type - break - dir_name = os.path.basename(root) - csv_filepath = os.path.join(root, dir_name + '.csv') - index_set = False - if os.path.exists(csv_filepath): - self.index_df = pd.read_csv(csv_filepath) - # filepath_csv = self.index_df.loc[0, 'filepath'] - # and os.path.dirname(filepath_csv) == os.path.dirname(raster_list[0]): - if len(self.index_df) == len(raster_list): - for _, row_df in self.index_df.iterrows(): - if crs is None: - crs = row_df['crs'] - if res is None: - res = row_df['res'] - # id = row_df['id'] - coords = (row_df['minx'], row_df['maxx'], - row_df['miny'], row_df['maxy'], - row_df['mint'], row_df['maxt']) - # change to relative path - filepath = os.path.join( - root, os.path.basename(row_df['filepath'])) - self.index.insert(i, coords, filepath) - i += 1 - index_set = True - print(f"Index loaded from: {os.path.basename(csv_filepath)}") - else: - print(f"Index file does not match the raster list, will be recreated.") - - if not index_set: - self.index_df = pd.DataFrame(columns=['id', - 'minx', 'maxx', 'miny', 'maxy', 'mint', 'maxt', - 'filepath', - 'crs', 'res']) - id_list = [] - coords_list = [] - filepath_list = [] - filename_regex = re.compile(self.filename_regex, re.VERBOSE) - # glob.iglob(pathname, recursive=True): - for filepath in raster_list: - match = re.match(filename_regex, os.path.basename(filepath)) - if match is not None: - try: - with rasterio.open(filepath) as src: - # See if file has a color map - if len(self.cmap) == 0: - try: - self.cmap = src.colormap(1) - except ValueError: - pass - - if crs is None: - crs = src.crs - if res is None: - res = src.res[0] - - with WarpedVRT(src, crs=crs) as vrt: - minx, miny, maxx, maxy = vrt.bounds - except rasterio.errors.RasterioIOError: - # Skip files that rasterio is unable to read - continue - else: - mint: float = 0 - maxt: float = sys.maxsize - if "date" in match.groupdict(): - date = match.group("date") - mint, maxt = disambiguate_timestamp( - date, self.date_format) - - coords = (minx, maxx, miny, maxy, mint, maxt) - self.index.insert(i, coords, filepath) - id_list.append(i) - coords_list.append(coords) - # change to relative path - filepath_list.append(os.path.basename(filepath)) - i += 1 - if i > 0: - self.index_df['id'] = id_list - self.index_df['filepath'] = filepath_list - self.index_df['minx'] = [coord[0] for coord in coords_list] - self.index_df['maxx'] = [coord[1] for coord in coords_list] - self.index_df['miny'] = [coord[2] for coord in coords_list] - self.index_df['maxy'] = [coord[3] for coord in coords_list] - self.index_df['mint'] = [coord[4] for coord in coords_list] - self.index_df['maxt'] = [coord[5] for coord in coords_list] - self.index_df.loc[:, 'crs'] = str(crs) - self.index_df.loc[:, 'res'] = res - index_set = True - self.index_df.to_csv(csv_filepath) - print(f"Index file: {os.path.basename(csv_filepath)} saved") - - if i == 0: - msg = f"No {self.__class__.__name__} data was found in `root='{self.root}'`" - if self.bands: - msg += f" with `bands={self.bands}`" - raise FileNotFoundError(msg) - - if not self.separate_files: - self.band_indexes = None - if self.bands: - if self.all_bands: - self.band_indexes = [ - self.all_bands.index(i) + 1 for i in self.bands - ] - else: - msg = ( - f"{self.__class__.__name__} is missing an `all_bands` " - "attribute, so `bands` cannot be specified." - ) - raise AssertionError(msg) - - self._crs = cast(CRS, crs) - self.res = cast(float, res) - - def __getitem__(self, query: Dict[str, Any]) -> Dict[str, Any]: - """Retrieve image/mask and metadata indexed by query. - - Args: - query: (minx, maxx, miny, maxy, mint, maxt) coordinates to index - - Returns: - sample of image/mask and metadata at that index - - Raises: - IndexError: if query is not found in the index - """ - # query may include the prompt points, bbox, or mask - bbox = query['bbox'] - filepath = query['path'] - - hits = self.index.intersection(tuple(bbox), objects=True) - filepaths = cast(List[str], [hit.object for hit in hits]) - - if not filepath in filepaths: - raise IndexError( - f"query: {bbox} not found in index with bounds: {self.bounds}" - ) - - if self.cache: - vrt_fh = self._cached_load_warp_file(filepath) - else: - vrt_fh = self._load_warp_file(filepath) - - # bounds = (bbox.minx, bbox.miny, bbox.maxx, bbox.maxy) - # band_indexes = self.band_indexes - - src = vrt_fh - dest = src.read() # read all bands - tags = src.tags() - if 'img_shape' in tags.keys(): - img_shape = tags['img_shape'] - input_shape = tags['input_shape'] - - # fix numpy dtypes which are not supported by pytorch tensors - if dest.dtype == np.uint16: - dest = dest.astype(np.int32) - elif dest.dtype == np.uint32: - dest = dest.astype(np.int64) - - tensor = torch.tensor(dest) # .float() - - # bbox may be useful to form the final mask results (geo-info) - sample = {"crs": self.crs, "bbox": bbox, "path": filepath} - if 'img_shape' in tags.keys(): - # convert string to python data structure - sample['img_shape'] = eval(img_shape) - sample['input_shape'] = eval(input_shape) - - if self.is_image: - sample["image"] = tensor.float() - else: - sample["mask"] = tensor # .float() #long() # modified zyzhao - - if self.transforms is not None: - sample = self.transforms(sample) - - return sample - - -class SamTestRasterDataset(RasterDataset): - filename_glob = "*.tif" - filename_regex = ".*" - date_format = "" - is_image = True - separate_files = False - all_bands = ['Red', 'Green', 'Blue', 'Alpha'] - rgb_bands = ['Red', 'Green', 'Blue'] - - def __init__(self, root: str = "data", - crs: Optional[CRS] = None, - res: Optional[float] = None, - bands: Optional[Sequence[str]] = None, - transforms: Optional[Callable[[ - Dict[str, Any]], Dict[str, Any]]] = None, - cache: bool = True - ) -> None: - if self.separate_files: - raise NotImplementedError( - 'Testing for separated files are not supported yet' - ) - super().__init__(root, crs, res, bands, transforms, cache) - - def __getitem__(self, query: Dict[str, Any]) -> Dict[str, Any]: - """Retrieve image/mask and metadata indexed by query. - - Args: - query: (minx, maxx, miny, maxy, mint, maxt) coordinates to index - - Returns: - sample of image/mask and metadata at that index - - Raises: - IndexError: if query is not found in the index - """ - bbox = query['bbox'] - filepath = query['path'] - patch_size = query['size'] - - hits = self.index.intersection(tuple(bbox), objects=True) - filepaths = cast(List[str], [hit.object for hit in hits]) - - if not filepath in filepaths: - raise IndexError( - f"query: {bbox} not found in index with bounds: {self.bounds}" - ) - - if self.cache: - vrt_fh = self._cached_load_warp_file(filepath) - else: - vrt_fh = self._load_warp_file(filepath) - - bounds = (bbox.minx, bbox.miny, bbox.maxx, bbox.maxy) - band_indexes = self.band_indexes - - src = vrt_fh - out_width = round((bbox.maxx - bbox.minx) / self.res) - out_height = round((bbox.maxy - bbox.miny) / self.res) - # out_width = math.ceil((bbox.maxx - bbox.minx) / self.res) - # out_height = math.ceil((bbox.maxy - bbox.miny) / self.res) - count = len(band_indexes) if band_indexes else src.count - out_shape = (count, out_height, out_width) - # if out_height == patch_size or out_width == patch_size: - # dest = src.read( - # indexes=band_indexes, - # out_shape=out_shape, - # window=window_from_bounds(*bounds, src.transform), - # ) - # else: - # resize - if out_height == 0 or out_width == 0: - raise Exception("Patch size should be greater than zero.") - - target_height, target_width = self.get_preprocess_shape( - out_height, out_width, patch_size) - target_shape = (count, target_height, target_width) - dest = src.read( - indexes=band_indexes, - out_shape=target_shape, - window=window_from_bounds(*bounds, src.transform), - resampling=Resampling.bilinear, - ) - - # fix numpy dtypes which are not supported by pytorch tensors - if dest.dtype == np.uint16: - dest = dest.astype(np.int32) - elif dest.dtype == np.uint32: - dest = dest.astype(np.int64) - - tensor = torch.tensor(dest) # .float() - if torch.isnan(tensor).any(): - # , posinf=0.0, neginf=0.0 - tensor = torch.nan_to_num(tensor, nan=0.0) - tensor = self.pad_patch(tensor, patch_size) - - sample = {"crs": self.crs, "bbox": bbox, - "path": filepath, "img_shape": out_shape, "input_shape": target_shape} - if self.is_image: - sample["image"] = tensor.float() - else: - sample["mask"] = tensor # .float() #long() # modified zyzhao - - if self.transforms is not None: - sample = self.transforms(sample) - - return sample - - @staticmethod - def get_preprocess_shape(old_h: int, old_w: int, long_side_length: int) -> Tuple[int, int]: - """ - Compute the output size given input size and target long side length. - """ - scale = long_side_length * 1.0 / max(old_h, old_w) - new_h, new_w = old_h * scale, old_w * scale - new_w = int(new_w + 0.5) # floor - new_h = int(new_h + 0.5) - return (new_h, new_w) - - @staticmethod - def pad_patch(x: Tensor, patch_size: int): - """ - Pad the patch to desired patch_size - """ - h, w = x.shape[-2:] - pad_h = patch_size - h - pad_w = patch_size - w - # pads are described starting from the last dimension and moving forward. - x = F.pad(x, (0, pad_w, 0, pad_h)) - return x - - def plot(self, sample, bright=1): - check_rgb = all(item in self.bands for item in self.rgb_bands) - if not check_rgb: - raise Exception("Need R G B bands to visualize") - - # Find the correct bands - rgb_indices = [] - for band in self.rgb_bands: - rgb_indices.append(self.bands.index(band)) - - # Reorder and rescale the image - if sample["image"].ndim == 4: - image = sample["image"][0, rgb_indices, :, :].permute(1, 2, 0) - else: - image = sample["image"][rgb_indices, :, :].permute(1, 2, 0) - - # if image.max() > 10: - # image = self.apply_scale(image) - image = torch.clamp(image*bright/255, min=0, max=1) - - # Plot the image - fig, ax = plt.subplots() - ax.imshow(image) - ax.axis('off') - - return fig - - -class SamTestFeatureGeoSampler(GeoSampler): - """Samples entire files at a time. - - This is particularly useful for datasets that contain geospatial metadata - and subclass :class:`~torchgeo.datasets.GeoDataset` but have already been - pre-processed into :term:`chips `. - - This sampler should not be used with :class:`~torchgeo.datasets.NonGeoDataset`. - You may encounter problems when using an :term:`ROI ` - that partially intersects with one of the file bounding boxes, when using an - :class:`~torchgeo.datasets.IntersectionDataset`, or when each file is in a - different CRS. These issues can be solved by adding padding. - """ - - def __init__( - self, - dataset: GeoDataset, - roi: Optional[BoundingBox] = None, - shuffle: bool = False, - ) -> None: - """Initialize a new Sampler instance. - - .. versionadded:: 0.3 - - Args: - dataset: dataset to index from - roi: region of interest to sample from (minx, maxx, miny, maxy, mint, maxt) - (defaults to the bounds of ``dataset.index``) - shuffle: if True, reshuffle data at every epoch - """ - - self.res = dataset.res - # self.dist_roi = ((feature_size*dataset.res/2)**2)*2 - self.dist_roi = None - self.q_bbox = None - self.q_path = None - if roi is None: - # self.index = dataset.index - # roi = BoundingBox(*self.index.bounds) - raise Exception('roi should be defined based on prompts!!!') - else: - self.index = Index(interleaved=False, - properties=Property(dimension=3)) - hits = dataset.index.intersection(tuple(roi), objects=True) - # hit_nearest = list(dataset.index.nearest(tuple(roi), num_results=1, objects=True))[0] - idx = 0 - for hit in hits: - idx += 1 - bbox = BoundingBox(*hit.bounds) # & roi - center_x_bbox = (bbox.maxx + bbox.minx)/2 - center_y_bbox = (bbox.maxy + bbox.miny)/2 - - center_x_roi = (roi.maxx + roi.minx)/2 - center_y_roi = (roi.maxy + roi.miny)/2 - dist_roi_tmp = (center_x_bbox - center_x_roi)**2 + \ - (center_y_bbox - center_y_roi)**2 - if idx == 1: - self.dist_roi = dist_roi_tmp - self.q_bbox = bbox - self.q_path = hit.object - elif dist_roi_tmp < self.dist_roi: - self.dist_roi = dist_roi_tmp - self.q_bbox = bbox - self.q_path = hit.object - # self.index.insert(hit.id, tuple(bbox), hit.object) - - print(f"Prompt intersected with {idx} feature patches") - - if self.q_bbox is None: - self.length = 0 - # raise Exception('no feature found intersected with prompts') - else: - self.length = 1 - self.roi = roi - - self.shuffle = shuffle - - def __iter__(self) -> Iterator[Dict[str, Any]]: - """Return the index of a dataset. - - Returns: - (minx, maxx, miny, maxy, mint, maxt) coordinates to index a dataset - and the exact single file filepath - """ - generator: Callable[[int], Iterable[int]] = range - if self.shuffle: - generator = torch.randperm - - for idx in generator(len(self)): - query = {"bbox": self.q_bbox, - "path": cast(str, self.q_path)} - yield query - - def __len__(self) -> int: - """Return the number of samples over the ROI. - - Returns: - number of patches that will be sampled - """ - return self.length # len(self.q_path) diff --git a/docs/Makefile b/docs/Makefile deleted file mode 100644 index d0c3cbf..0000000 --- a/docs/Makefile +++ /dev/null @@ -1,20 +0,0 @@ -# Minimal makefile for Sphinx documentation -# - -# You can set these variables from the command line, and also -# from the environment for the first two. -SPHINXOPTS ?= -SPHINXBUILD ?= sphinx-build -SOURCEDIR = source -BUILDDIR = build - -# Put it first so that "make" without argument is like "make help". -help: - @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) - -.PHONY: help Makefile - -# Catch-all target: route all unknown targets to Sphinx using the new -# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). -%: Makefile - @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/__init__.py b/docs/__init__.py deleted file mode 100644 index 1af9299..0000000 --- a/docs/__init__.py +++ /dev/null @@ -1,3 +0,0 @@ -import os - -encoder_help = os.path.dirname(__file__) + '/encoder.help' diff --git a/docs/encoder.help b/docs/encoder.help deleted file mode 100644 index 317df6d..0000000 --- a/docs/encoder.help +++ /dev/null @@ -1,13 +0,0 @@ -This tool helps to preprocess geospatial images and generate image features using the SAM image encoder. The generated image features can then be used in our Geo-SAM tool to label the landforms by adding points and bounding box prompts. - -After selecting the raster layer or image file you want to process, you should also choose the proper bands. The SAM natively supports only three-band RGB images, but we have adapted the tool to support one or two-band images so that you can try grayscale images or NDVI spectral index images. - -Since SAM only supports input images with size of (1024, 1204), small images will be resized to match the input size, while large images will be sampled into overlapped patches (patch_size=1024) in a grid-like fashion. The stride parameter will determine the overlap behavior, overlap = patch_size - stride. - -The values of the image input to the SAM should range from 0 to 255, and you may need to specify the value range to be rescaled to [0, 255]. By default, the tool will help you to find the min and max values of the image data and rescale the value range of [min, max] to [0, 255]. - -SAM model checkpoints should be downloaded in advance, and three versions (huge, large, base) are available. The large version "vit_l" is recommended to try first. You need to specify the model type that matches the checkpoint version. - -Tips for making encoding process faster: - -1. Choose smaller processing extent; 2. Reduce target resolution; 3. Increase stride to minimize overlap; 4. Choose a smaller version of sam model; 5. Use GPU; 6. Increase batch_size when using a GPU with sufficient gpu memory; diff --git a/docs/requirements.txt b/docs/requirements.txt deleted file mode 100644 index 85dfaae..0000000 --- a/docs/requirements.txt +++ /dev/null @@ -1,6 +0,0 @@ -recommonmark -sphinx==3.5.3 -myst-parser -sphinxcontrib-video -sphinx_rtd_theme -Jinja2<3.1 diff --git a/docs/source/Acknowledgement.md b/docs/source/Acknowledgement.md deleted file mode 100644 index 4626a0c..0000000 --- a/docs/source/Acknowledgement.md +++ /dev/null @@ -1,3 +0,0 @@ -# Acknowledgement - -This repo benefits from [Segment Anything](https://github.com/facebookresearch/segment-anything) and [TorchGeo](https://github.com/microsoft/torchgeo). Thanks for their wonderful work. diff --git a/docs/source/Citation.md b/docs/source/Citation.md deleted file mode 100644 index 5e26246..0000000 --- a/docs/source/Citation.md +++ /dev/null @@ -1,17 +0,0 @@ - -# Citation - -> Zhao, Zhuoyi, Fan, Chengyan, & Liu, Lin. (2023). Geo SAM: A QGIS plugin using Segment Anything Model (SAM) to accelerate geospatial image segmentation (1.1.0). Zenodo. - -```bibtex -@software{zhao_zhuoyi_2023_8191039, - author = {Zhao, Zhuoyi and Fan, Chengyan and Liu, Lin}, - title = {{Geo SAM: A QGIS plugin using Segment Anything Model (SAM) to accelerate geospatial image segmentation}}, - month = jul, - year = 2023, - publisher = {Zenodo}, - version = {1.1.0}, - doi = {10.5281/zenodo.8191039}, - url = {https://doi.org/10.5281/zenodo.8191039} -} -``` \ No newline at end of file diff --git a/docs/source/Installation.md b/docs/source/Installation.md deleted file mode 100644 index e99a5fd..0000000 --- a/docs/source/Installation.md +++ /dev/null @@ -1,126 +0,0 @@ - -# Installation - -## Install QGIS - -You are suggested to install the latest version of [QGIS](https://www.qgis.org/en/site/forusers/download.html) since the plugin has mainly been tested on versions newer than QGIS 3.30 (QGIS 3.28 LTR should also work fine). - -## Install Library Dependencies - -Some dependencies need to be installed into the Python environment in QGIS beforehand to use Geo-SAM. `Pytorch` is a fundamental dependency. If you want to install the GPU version of `Pytorch`, it is recommended to refer to the official website for installation: - -After installing `PyTorch`, `torchgeo` and `segment-anything` need to be installed subsequently. Below are tutorials for installing these dependencies on different operating systems. - -### For Windows Users - -Open the **OSGeo4W Shell** ![OsGeo4WShell](img/OsGeo4WShell.png) application from the Start menu, which is a dedicated shell for the QGIS. Then run the following command to install the libraries. - -```bash -pip3 install torch torchvision -pip3 install torchgeo -pip3 install segment-anything -``` - -`Geo-SAM Encoder Tool` now supports using CUDA GPU to accelerate the encoding process. If your PC has dedicated CUDA GPUs, you can install the CUDA library first and then install the gpu-version pytorch using the following command (using CUDA version 11.7 as an example): - -```bash -pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117 -# or if you have installed the CPU version before. -pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117 --force-reinstall -``` - -### For Mac or Linux Users - -Open your own terminal application, and change the directory to where the QGIS Python binary file locates. - -```bash -# Mac -cd /Applications/QGIS.app/Contents/MacOS/bin -# Linux (Please confirm the python env by "import qgis") -cd /usr/bin -``` - -To confirm the QGIS Python environment: - -```bash -./python3 ->>> import qgis -``` - -Then install the libraries. - -```bash -# !important, add ./ to avoid using your default Python in the system -./pip3 install torch torchvision -./pip3 install torchgeo -./pip3 install segment-anything -``` - -For Linux users, if `pip3` is not found in `/usr/bin`, try the following commands: - -```bash -sudo apt-get update -sudo apt-get install python3-pip -``` - -For Linux users, if your computer got available CUDA GPUs and with CUDA library installed, the above commands should have helped you install the gpu-version pytorch. You can reach [pytorch official website](https://pytorch.org/get-started/locally/) for more information. - -## Install the Geo-SAM Plugin - -### Download the Plugin - -Download the `stable version`: [plugin zip file](https://github.com/coolzhao/Geo-SAM/releases/tag/v1.1.1), or the `dev version` (more features and capabilities, but not rigorous tested): [plugin zip file](https://github.com/coolzhao/Geo-SAM/releases/tag/v1.2.1-dev), unzip it, and rename the folder as `Geo-SAM` (be aware of undesired nested folders after unzipping). - - -### Locate the QGIS Plugin folder - -In QGIS, Go to the menu `Settings` > `User Profiles` > `Open active profile folder.` You'll be taken straight to the profile directory. Under the profile folder, you may find a `python` folder; the `plugins` folder should be right inside the `python` folder (create the `plugins` folder if it does not exist). Put the entire `Geo-SAM` folder inside the `plugins` folder, then restart QGIS. The directory tree structure should be the same as the following. - -```bash -python -└── plugins - └── Geo-SAM -    ├── checkpoint -    ├── docs -    ├── ... -    ├── tools -    └── ui -``` - -Below are some general paths of the plugin folder for your reference. - -```bash -# Windows -%APPDATA%\QGIS\QGIS3\profiles\default\python\plugins -# Mac -~/Library/Application\ Support/QGIS/QGIS3/profiles/default/python/plugins -# Linux -~/.local/share/QGIS/QGIS3/profiles/default/python/plugins -``` - -### Activate the Geo-SAM Plugin - -After restarting QGIS, go to the menu `Plugins` > `Manage and Install Plugins`, and under `Installed`, you may find the `Geo SAM` plugin; check it to activate the plugin. - - -```{image} img/Active_geo_sam.png -:alt: Plugin menu -:width: 90% -:align: center -``` - -After activating the Geo-SAM plugin, you may find the Geo SAM tools under the `Plugins` menu, - -```{image} img/Plugin_menu_geo_sam.png -:alt: Plugin menu -:width: 60% -:align: center -``` - -You may also find a new toolbar, including three icons. - -```{image} img/Toolbar_geo_sam.png -:alt: Plugin toolbar -:width: 33% -:align: center -``` diff --git a/docs/source/Usage/copilot.rst b/docs/source/Usage/copilot.rst deleted file mode 100644 index 28793ee..0000000 --- a/docs/source/Usage/copilot.rst +++ /dev/null @@ -1,47 +0,0 @@ -Encoder Copilot (QGIS plugin) -============================= - -Background ----------- - -SAM only supports 1024x1024 pixel image inputs, but the pixels in our remote sensing images are generally much greater than this number. To solve this problem, we split the image into overlapping patches. After the user provides prompts, the patch which center closest to the center of the prompts is selected for SAM execution. However, if the spatial resolution of the image is too high or the area of the land surface objects is too large, 1024x1024 may not be able to cover those objects. To avoid this, the spatial resolution of image needs to be reduced. The ``Geo-SAM Image Encoder`` tool supports changing the output image resolution, but we usually don't know which value of resolution is appropriate. To allow the user to check in real time whether the patch size covers the desired objects under different resolutions, we designed the ``Encoder Copilot`` tool to display all patches under different resolutions and overlaps/strides in real time. - -Usage ------ - -Click the ``Encoder Copilot`` icon to open the interactive Encoder Copilot widget. Select the raster and bands you want to encode, and then set the extent to be processed. The extent can be calculated from the layer or drawn directly on the canvas. At this point, adjusting the resolution scale and overlap values will display the corresponding patches situation in real time. - -Besides, ``Encoder Copilot`` also supports direct setting of other parameters. You can specify the path of the SAM model checkpoint, and the model type will be automatically analyzed from the file name. If you change the file name of the checkpoint, the automatically detected model type may not be correct, you need to select it manually in this case. The value range supported by the SAM model is 0-255, but the value range of remote sensing images is usually not within this range. Our tool will automatically detect the value range of the image and stretch it to 0-255. Users can click on the “parse from raster” button in Copilot to directly detect the value range, or manually modify this value. If you do nothing, don’t worry, ``Geo-SAM Image Encoder`` tool can also automatically detect and stretch the value range. - -Export Settings ---------------- - -Copy Settings to clipboard -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When the parameters are determined, the user can directly click the ``Copy Setting`` button in ``Encoder Copilot``. Then click ``Advanced`` -> ``Paste Settings`` in the ``Geo-SAM Image Encoder`` to paste the parameters into the tool. - -.. image:: ../img/Paste_Settings.jpg - :alt: Paste_Settings - :width: 600px - :align: center - - -Export Settings into a file -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Users can also export the settings to a json file by click ``Export Settings`` button. This is useful to pass parameters to the :ref:`GeoSAM-Image-Encoder ` package. - -Demo Animation --------------- - -The following YouTube video shows how to use the ``Encoder Copilot``. - -.. image:: ../_static/EncoderCopilotCover.jpg - :alt: EncoderCopilotDemo - :width: 95% - :target: https://youtu.be/NWemi3xcCd0 - - - - diff --git a/docs/source/Usage/encoder_package.rst b/docs/source/Usage/encoder_package.rst deleted file mode 100644 index 4f74836..0000000 --- a/docs/source/Usage/encoder_package.rst +++ /dev/null @@ -1,136 +0,0 @@ -.. _Geo-SAM Image Encoder: - -GeoSAM-Image-Encoder (Python package) -====================================== - -.. image:: https://img.shields.io/pypi/v/GeoSAM-Image-Encoder - :target: https://pypi.org/project/GeoSAM-Image-Encoder/ - - - -This package is part of the Geo-SAM project and is a standalone Python package that does not depend on QGIS. This package allows you to encode remote sensing images into features that can be recognized by Geo-SAM on a remote server, such as ``Colab`` or ``AWS``. - - -Installation ------------- -.. note:: - - Installing ``GeoSAM-Image-Encoder`` directly will install the CPU version of ``PyTorch``. Therefore, it is recommended to install the appropriate version of ``PyTorch`` before installing ``GeoSAM-Image-Encoder`` in your machine. You can install the corresponding version based on the official ``PyTorch`` website: https://pytorch.org/get-started/locally/#start-locally - -After installing PyTorch, you can install ``GeoSAM-Image-Encoder`` via pip. - -.. code-block:: bash - - pip install GeoSAM-Image-Encoder - # or - pip install git+https://github.com/Fanchengyan/GeoSAM-Image-Encoder.git - - -Usage ------ - -You can call this script in Python or Terminal. We recommend using Python interface directly for processing, which will have greater flexibility. - -Using Python -~~~~~~~~~~~~ - -After install GeoSAM-Image-Encoder, you can import it using ``geosam`` - -.. code-block:: python - - import geosam - from geosam import ImageEncoder - -check the folder contains geosam and add it to environment if you want to run in terminal - -.. code-block:: python - - geosam.folder - -check if gpu available - - -.. code-block:: python - - geosam.gpu_available() - -Run by specify parameters directly -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. code-block:: python - - checkpoint_path = '/content/sam_vit_l_0b3195.pth' - image_path = '/content/beiluhe_google_img_201211_clip.tif' - feature_dir = './' - - ## init ImageEncoder - img_encoder = ImageEncoder(checkpoint_path) - ## encode image - img_encoder.encode_image(image_path,feature_dir) - - -Run by parameters from setting.json file -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If you want to using settings.json file to provide the parameters - -.. code-block:: python - - setting_file = "/content/setting.json" - feature_dir = './' - - ### parse settings from the setting,json file - settings = geosam.parse_settings_file(setting_file) - - ### setting file not contains feature_dir, you need add it - settings.update({"feature_dir":feature_dir}) - - ### split settings into init_settings, encode_settings - init_settings, encode_settings = geosam.split_settings(settings) - - print(f"settings: {settings}") - print(f"init_settings: {init_settings}") - print(f"encode_settings: {encode_settings}") - -Then, you can run image encoding by parameters from setting.json file - -.. code-block:: python - - img_encoder = ImageEncoder(**init_settings) - img_encoder.encode_image(**encode_settings) - -Using Terminal -~~~~~~~~~~~~~~ - -check the folder of geosam - -.. code-block:: bash - - print(geosam.folder) - -add this folder into environment of your machine. Then run in terminal: - -.. code-block:: bash - - image_encoder.py -i /content/beiluhe_google_img_201211_clip.tif -c /content/sam_vit_l_0b3195.pth -f ./ - -You can overwrite the settings from file by specify the parameter values. For Example - -.. code-block:: bash - - image_encoder.py -s /content/setting.json -f ./ --stride 256 --value_range "10,255" - -check all available parameters: - -.. code-block:: bash - - image_encoder.py -h - - -Colob Example -------------- - - -You can click on the link below to experience GeoSAM-Image-Encoder in ``Colab``: - -``_ diff --git a/docs/source/Usage/encoding.md b/docs/source/Usage/encoding.md deleted file mode 100644 index 0aa729b..0000000 --- a/docs/source/Usage/encoding.md +++ /dev/null @@ -1,43 +0,0 @@ - -# Geo-SAM Image Encoder (QGIS plugin) - -If you want to try your own images, you can use the `Geo-SAM Image Encoder` tool. This tool helps to preprocess geospatial images and generate image features using the SAM image encoder. The generated image features can then be used in our `Geo-SAM Segmentation` tool to label the landforms by adding points and bounding box prompts. - -## Download SAM Checkpoints - -SAM model checkpoints should be downloaded in advance, and three versions (huge, large, and base) are available. The large version "vit_l" is recommended to try first. You need to specify the model type that matches the checkpoint version. Using the following links to download the checkpoints. - -- `vit_h`: [ViT-H SAM model.](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth) -- `vit_l`: [ViT-L SAM model.](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_l_0b3195.pth) -- `vit_b`: [ViT-B SAM model.](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth) - -## Select Bands and Value Range for Processing - -After selecting the raster layer or image file you want to process, you should also choose the proper bands. The SAM natively supports only three-band RGB images, but we have adapted the tool to support one or two-band images so that you can try grayscale images or NDVI spectral index images. - -The values of the image input to the SAM should range from 0 to 255, and you may need to specify the value range (in `Advanced Parameters`) to be rescaled to [0, 255]. By default, the tool will help you to find the min and max values of the image data and rescale the value range of [min, max] to [0, 255]. - -## Patch Sampling - -Since SAM only supports input images with sizes of (1024, 1204), small images will be resized to match the input size, while large images will be sampled into overlapped patches (patch_size=1024) in a grid-like fashion. The stride parameter will determine the overlap behavior, overlap = patch_size - stride. - -## Demo Animation - -The following animation shows how to use the encoding tool. - -```{image} ../img/encoder_demo.gif -:alt: Try Geo SAM -:width: 600px -:align: center -``` - -After processing the image, by default, the generated features will automatically be loaded in the segmentation tool for you to start labeling. Or you can choose to load the image features manually afterward. - -## Tips for Making the Encoding Process Faster - -- Choose a smaller processing extent -- Reduce target resolution (in `Advanced Parameters`) -- Increase stride to minimize overlap -- Choose a smaller version of SAM model -- Use GPU -- Increase batch_size when using a GPU with sufficient GPU memory diff --git a/docs/source/Usage/index.rst b/docs/source/Usage/index.rst deleted file mode 100644 index d9e88dd..0000000 --- a/docs/source/Usage/index.rst +++ /dev/null @@ -1,13 +0,0 @@ -Usage of Geo-SAM Tools -====================== - -This section describes the usage of the tools provided in ``Geo-SAM``. - -.. toctree:: - :maxdepth: 1 - - segmentation - encoding - encoder_package - copilot - diff --git a/docs/source/Usage/segmentation.rst b/docs/source/Usage/segmentation.rst deleted file mode 100644 index f880abf..0000000 --- a/docs/source/Usage/segmentation.rst +++ /dev/null @@ -1,154 +0,0 @@ -Geo-SAM Segmentation (QGIS plugin) -================================== - -Click the ``Geo-SAM Segmentation`` icon to open the interactive segmentation widget. You will be shown a demo raster image with thaw slump and small pond landforms for you to try the tool. With a single click on the map, a segmentation result will be generated. - -.. image:: ../img/try_geo_sam.gif - :alt: try_geo_sam - :width: 500px - :align: center - -A user interface will be shown below. - -.. image:: ../img/ui_geo_sam.png - :alt: Geo SAM UI - :width: 600px - :align: center - - -Prompt Tab ----------- - -Load Image Features -~~~~~~~~~~~~~~~~~~~ - -The plugin is initialized with features for demo purposes, and you can use the ``Feature Folder`` selection button to select the folder that includes the image features you need. - -.. image:: ../img/Select_feature_folder.png - :alt: Select feature folder - :width: 250px - :align: center - - -Then, press the ``Load`` button to load the selected image features. Remember to add the corresponding raster image to the QGIS project. - -SAM Output Features -~~~~~~~~~~~~~~~~~~~ - -The Shapefile used to save the segmentation results. You can specify it from: - -- the current layers in QGIS project (only support previous saved Shapefile using this plugin) -- a file (will be created if not exist). - -Add Prompts -~~~~~~~~~~~ - -There are 3 types of prompts: - -- ``Foreground Point (FG)``: a point that indicates the foreground of the object (desired area) -- ``Background Point (BG)``: a point that indicates the background of the object (unwanted area) -- ``Bounding Box (BBox)``: a bounding box that limits the boundary of the object - -.. note:: - - - You can add multiple prompts for each type and combine them to segment one object. - - You can use the ``Tab`` button to loop between the 3 prompt types. The cursor will also change to the corresponding types. - - -Undo/Clear Prompts -~~~~~~~~~~~~~~~~~~ - -- Press ``Undo`` button (Shortcut: ``Z``) to undo the last prompt. -- Press ``Clear`` button (Shortcut: ``C``) to clear all prompts and unsaved segmentation results. - -Minimum Pixels -~~~~~~~~~~~~~~ - -The minimum pixels for the segmentation result. The object with fewer pixels than the ``Minimum Pixels`` value will be removed from the segmentation results. The ``Minimum Pixels`` will change to ``Default Minimum Pixels`` after saved the segmentation results. - -Save Current Results -~~~~~~~~~~~~~~~~~~~~ - -You can save the segmentation results by clicking the ``Save`` button (Shortcut: ``S``), after adding points and a BBox prompts for segmenting a object. - - - -Preview mode -~~~~~~~~~~~~ - -To facilitate better labeling, we have developed a preview mode. In this mode, SAM's execution results are displayed ``in real-time as you move the mouse``, allowing to observe the prompt's effect instantly. In this mode, users can choose the most suitable prompt without worrying about getting worse by adding new prompt, reducing the frequency of prompt modifications. - -.. note:: - - - The ``Preview mode`` is only designed for previewing purposes, and only the ``pressed prompts`` will be saved. - - You can use the ``P`` key to toggle the ``Preview mode`` on/off. - - -.. image:: ../img/PreviewModeDemo.gif - :alt: preview_mode - :width: 500px - :align: center - - -Enable/Disable the Tool -~~~~~~~~~~~~~~~~~~~~~~~ - -You can uncheck the ``Enable`` button to temporally disable the tool and navigate on the map. - - - -Shortcuts -~~~~~~~~~ - -- ``Tab``: loop between 3 prompt types (the cursor will also change to the corresponding types) -- ``P``: Toggle to enable/disable executing SAM with ``Preview mode`` -- ``C``: clear all prompts in canvas [same as ``Clear`` button] -- ``Z``: undo the last prompt in canvas [same as ``Undo`` button] -- ``S``: save SAM output features into polygon [same as ``Save`` button] -- ``Ctrl+Z`` or ``command+Z``: undo the last saved segmentation results - -Tips for Using the Segmentation Tool -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -- Deal with only **One object** each time -- Use **Background Points** to exclude unwanted parts -- Use **Bounding Box (BBox)** to limit the segment polygon boundary -- The **BBox** should cover the entire object -- Remember to press the ``Save`` button after the segmentation of the chosen object - -Settings Tab ------------- - -Colors -~~~~~~ - -Due to the diverse range of colors in remote sensing images, the default color scheme may closely resemble the image colors, making it difficult to differentiate. Therefore, we have added an option to modify colors in the Settings Tab. - -There are 3 types of colors for the Prompts: - -- ``Foreground Point``: the color of the foreground point -- ``Background Point``: the color of the background point -- ``Bounding Box(BBox)``: the color of the bounding box - -There are 3 types of colors for the segmentation results: - -- ``Shapefile Polygon``: the color of the segmentation results saved in the Shapefile. You can change this color by changing the layer color in QGIS directly. -- ``Prompt Polygon``: the color of the segmentation results from pressed prompts -- ``Preview Polygon``: the color of the segmentation results in the preview mode (results that with mouse moving) - -The ``Boundary`` color is the color of the Boundary of the image feature. - -Load Demo -~~~~~~~~~ - -enable/disable loading demos when open widget. - -Show Boundary -~~~~~~~~~~~~~ - -enable/disable showing the boundary of the image feature in the canvas. - -Default Minimum Pixels -~~~~~~~~~~~~~~~~~~~~~~ - -The default minimum pixels for the segmentation result. The object with fewer pixels than the default value will be removed from the segmentation results. The ``Minimum Pixels`` will change to ``Default Minimum Pixels`` after saved the segmentation results. \ No newline at end of file diff --git a/docs/source/_static/EncoderCopilotCover.jpg b/docs/source/_static/EncoderCopilotCover.jpg deleted file mode 100644 index 29f60ea..0000000 Binary files a/docs/source/_static/EncoderCopilotCover.jpg and /dev/null differ diff --git a/docs/source/_static/EncoderCopilotDemo.mp4 b/docs/source/_static/EncoderCopilotDemo.mp4 deleted file mode 100644 index b0a82a0..0000000 Binary files a/docs/source/_static/EncoderCopilotDemo.mp4 and /dev/null differ diff --git a/docs/source/conf.py b/docs/source/conf.py deleted file mode 100644 index d3dcd53..0000000 --- a/docs/source/conf.py +++ /dev/null @@ -1,41 +0,0 @@ -# Configuration file for the Sphinx documentation builder. -# -# For the full list of built-in configuration values, see the documentation: -# https://www.sphinx-doc.org/en/master/usage/configuration.html - -# -- Project information ----------------------------------------------------- -# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information - -project = 'Geo-SAM' -copyright = '2023, Joey, Fancy' -author = 'Joey, Fancy' -release = 'v1.2' - -# -- General configuration --------------------------------------------------- -# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration - -extensions = [ - 'myst_parser', - 'sphinx.ext.autodoc', - 'sphinxcontrib.video', - 'sphinx.ext.doctest', - 'sphinx.ext.intersphinx', - 'sphinx.ext.napoleon', - 'sphinx.ext.todo', -] -source_suffix = { - '.rst': 'restructuredtext', - '.md': 'markdown', -} - -# templates_path = ['_templates'] -exclude_patterns = [] - - -# -- Options for HTML output ------------------------------------------------- -# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output - -html_theme = 'sphinx_rtd_theme' -html_static_path = ['_static'] - -video_enforce_extra_source = True diff --git a/docs/source/future.md b/docs/source/future.md deleted file mode 100644 index 8224bae..0000000 --- a/docs/source/future.md +++ /dev/null @@ -1,6 +0,0 @@ -# Future Works - -- Support HQ-SAM model -- Support labeling large objects across patches -- Add video tutorials for installation and usage - diff --git a/docs/source/img/.$Geo_SAM.drawio.bkp b/docs/source/img/.$Geo_SAM.drawio.bkp deleted file mode 100644 index 585c574..0000000 --- a/docs/source/img/.$Geo_SAM.drawio.bkp +++ /dev/null @@ -1,188 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/docs/source/img/Active_geo_sam.png b/docs/source/img/Active_geo_sam.png deleted file mode 100644 index 9931d65..0000000 Binary files a/docs/source/img/Active_geo_sam.png and /dev/null differ diff --git a/docs/source/img/Geo_SAM.drawio b/docs/source/img/Geo_SAM.drawio deleted file mode 100644 index 789d2f0..0000000 --- a/docs/source/img/Geo_SAM.drawio +++ /dev/null @@ -1,185 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/docs/source/img/Geo_SAM.png b/docs/source/img/Geo_SAM.png deleted file mode 100644 index d74d7c0..0000000 Binary files a/docs/source/img/Geo_SAM.png and /dev/null differ diff --git a/docs/source/img/Load_image_feature.png b/docs/source/img/Load_image_feature.png deleted file mode 100644 index ebb01f9..0000000 Binary files a/docs/source/img/Load_image_feature.png and /dev/null differ diff --git a/docs/source/img/OsGeo4WShell.png b/docs/source/img/OsGeo4WShell.png deleted file mode 100644 index 165fb31..0000000 Binary files a/docs/source/img/OsGeo4WShell.png and /dev/null differ diff --git a/docs/source/img/Paste_Settings.jpg b/docs/source/img/Paste_Settings.jpg deleted file mode 100644 index 2433cbd..0000000 Binary files a/docs/source/img/Paste_Settings.jpg and /dev/null differ diff --git a/docs/source/img/Plugin_menu_geo_sam.png b/docs/source/img/Plugin_menu_geo_sam.png deleted file mode 100644 index 24143a8..0000000 Binary files a/docs/source/img/Plugin_menu_geo_sam.png and /dev/null differ diff --git a/docs/source/img/PreviewModeDemo.gif b/docs/source/img/PreviewModeDemo.gif deleted file mode 100644 index 5dcdaec..0000000 Binary files a/docs/source/img/PreviewModeDemo.gif and /dev/null differ diff --git a/docs/source/img/Select_feature_folder.png b/docs/source/img/Select_feature_folder.png deleted file mode 100644 index 4c2e3f8..0000000 Binary files a/docs/source/img/Select_feature_folder.png and /dev/null differ diff --git a/docs/source/img/Toolbar_geo_sam.png b/docs/source/img/Toolbar_geo_sam.png deleted file mode 100644 index ee47bc0..0000000 Binary files a/docs/source/img/Toolbar_geo_sam.png and /dev/null differ diff --git a/docs/source/img/encoder_demo.gif b/docs/source/img/encoder_demo.gif deleted file mode 100644 index d94128d..0000000 Binary files a/docs/source/img/encoder_demo.gif and /dev/null differ diff --git a/docs/source/img/try_geo_sam.gif b/docs/source/img/try_geo_sam.gif deleted file mode 100644 index 0b406e2..0000000 Binary files a/docs/source/img/try_geo_sam.gif and /dev/null differ diff --git a/docs/source/img/try_geo_sam.png b/docs/source/img/try_geo_sam.png deleted file mode 100644 index 618a711..0000000 Binary files a/docs/source/img/try_geo_sam.png and /dev/null differ diff --git a/docs/source/img/ui_geo_sam.png b/docs/source/img/ui_geo_sam.png deleted file mode 100644 index 8d58c63..0000000 Binary files a/docs/source/img/ui_geo_sam.png and /dev/null differ diff --git a/docs/source/index.rst b/docs/source/index.rst deleted file mode 100644 index 799658b..0000000 --- a/docs/source/index.rst +++ /dev/null @@ -1,51 +0,0 @@ -.. Geo-SAM documentation master file, created by - sphinx-quickstart on Sun Aug 13 16:07:25 2023. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - -=================================== -Welcome to Geo-SAM's documentation! -=================================== - -By `Joey `_ and `Fancy `_ from `Cryosphere Lab `_, ESSC, CUHK. - -Introduction ------------- - -Geo-SAM is a QGIS plugin that aims to help people segment, delineate or label landforms efficiently when using large-size geospatial raster images. `Segment Anything Model `_ (SAM) is a foundation AI model with the superpower, but the model size is huge, and using it to process images can take a long time, even with a modern GPU. Our tool uses the strategies of encoding image features in advance and trimming the SAM model. **The interactive segmentation process can be run in real-time (at** ``millisecond`` **speeds) on a laptop by only using a CPU**, making it a convenient and efficient tool for dealing with remote sensing images. - -The Geo-SAM plugin includes two separate parts: the ``Image Encoding Part``, and the ``Interactive Segmentation Part``. The image encoding part is designed to generate and save the image features using the SAM image encoder, and the encoding process only needs to run once per image. The segmentation part is for interactively segmenting landforms, and it can only be used to segment preprocessed images (whose features have been generated in advance using the encoding tool, as the included demo image). - - -.. figure:: img/Geo_SAM.png - :width: 100% - :align: center - :alt: Geo-SAM - - Comparison of the workflow between Geo-SAM and the original SAM. The original SAM model encodes prompts and image simultaneously, while the Geo-SAM model encodes image into feature files at once and queries prompts in real-time (at ``millisecond`` speeds) by loading those saved features. - -Reasons for choosing Geo-SAM ----------------------------- - -* Based on QGIS for a user-friendly GUI and cross-platform compatibility without programming skills needed. -* It provides segmentation results ``instantly after giving prompts``, and can even display results ``in real-time following the mouse cursor`` (Preview mode, currently only available in the dev version, will be added to the stable version after being rigorously tested). Users can have a smooth, interactive experience.This can greatly improve the efficiency and user experience of segmentation. - - -.. note:: - - - SAM is designed to **segment one object once with a series of prompts**, so you should save the current results before getting to the next one when using the Geo-SAM tool. - - SAM natively supports only three-band images, but we have adapted Geo-SAM to support one or two-band images so that you can try grayscale images, spectral index images (like NDVI, NDWI), or even SAR images. - - The Geo-SAM plugin is currently in active development. We will continue making improvements and welcome your feedback. If you have any questions or suggestions, please feel free to open an issue on our GitHub repository at `GitHub issue `_. - - -.. toctree:: - :maxdepth: 2 - :caption: Contents: - - Installation - Usage/index - future - Citation - Acknowledgement - -