1688 Data

Name: 1688 Data
Rating: 3.5 (1 reviews)
Author: aoyasong

By aoyasong 👁 10 views ▲ 0 votes

OpenClaw MCP Plugin for 1688 platform product data and supplier information

GitHub

Install

npm install
```

Configuration Example

authentication:
  api_key: "your_key"
  api_secret: "your_secret"
  # OR
  cookie: "your_cookie"

request:
  timeout: 30000
  max_retries: 3

scraping:
  use_playwright: false
  headless: true

README

# 1688 Data Plugin for OpenClaw

OpenClaw MCP Plugin for 1688 platform (Alibaba procurement wholesale) product data collection and supplier information query.

## Overview

This plugin provides tools to search and retrieve product data from 1688.com (Alibaba's wholesale platform), including:
- Product search and filtering
- Detailed product information (price, specifications, stock, images)
- Supplier information and ratings
- Price history analysis
- Market trend analysis
- Minimum order quantity (MOQ) retrieval

## Features

- **Multiple Authentication Methods**: Supports API keys, OAuth tokens, or web scraping with cookies
- **Anti-Scraping Measures**: Built-in rate limiting, random delays, and user-agent rotation
- **Proxy Support**: Optional proxy configuration for IP rotation
- **Playwright Integration**: Optional headless browser for JavaScript-heavy pages
- **Caching Layer**: In-memory cache to reduce redundant requests
- **Data Normalization**: Standardized output compatible with ecommerce-db-plugin
- **Error Handling**: Robust retry mechanisms and error recovery

## Prerequisites

- Node.js v18 or higher
- OpenClaw gateway with MCP support
- 1688 account (for web scraping) or 1688 Open Platform API credentials

## Installation

### 1. Clone the repository

```bash
git clone https://github.com/aoyasong/1688-data-plugin.git
cd 1688-data-plugin
```

### 2. Install dependencies

```bash
npm install
```

### 3. Configure authentication

Choose one authentication method and configure it:

#### Option A: Web scraping with cookie (recommended for testing)

1. Log in to 1688.com in your browser
2. Copy the cookie value (use browser developer tools)
3. Create `.env` file:

```bash
cp .env.example .env
```

4. Edit `.env` and add your cookie:

```env
ALIBABA_COOKIE="your_1688_cookie_here"
```

#### Option B: 1688 Open Platform API (if available)

```env
ALIBABA_API_KEY="your_api_key"
ALIBABA_API_SECRET="your_api_secret"
ALIBABA_ACCESS_TOKEN="your_access_token"  # Optional
```

#### Option C: Proxy configuration (optional)

```env
PROXY_URL="http://127.0.0.1:7890"
```

### 4. Test the plugin

```bash
# Run basic usage example
node examples/basic-usage.js

# Run crawl test
node examples/crawl-test.js
```

## Configuration

### Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `ALIBABA_API_KEY` | 1688 Open Platform API key | - |
| `ALIBABA_API_SECRET` | 1688 Open Platform API secret | - |
| `ALIBABA_ACCESS_TOKEN` | OAuth access token | - |
| `ALIBABA_REFRESH_TOKEN` | OAuth refresh token | - |
| `ALIBABA_COOKIE` | 1688 website cookie for scraping | - |
| `PROXY_URL` | HTTP/HTTPS proxy URL | - |
| `USE_PLAYWRIGHT` | Use Playwright for JavaScript-heavy pages | `false` |
| `RATE_LIMIT_REQUESTS_PER_MINUTE` | Rate limit for requests | `10` |
| `CACHE_TTL_SECONDS` | Cache time-to-live in seconds | `900` |
| `CACHE_MAX_ITEMS` | Maximum cache items | `100` |
| `LOG_LEVEL` | Logging level (`debug`, `info`, `warn`, `error`) | `info` |

### YAML Configuration

You can also use a YAML configuration file at `config/alibaba.yaml`:

```yaml
authentication:
  api_key: "your_key"
  api_secret: "your_secret"
  # OR
  cookie: "your_cookie"

request:
  timeout: 30000
  max_retries: 3

scraping:
  use_playwright: false
  headless: true
```

## Integration with OpenClaw

### Add to OpenClaw configuration

Add the plugin to your OpenClaw configuration:

```yaml
# openclaw.yaml
plugins:
  - type: mcp
    name: 1688-data-plugin
    config:
      command: node
      args:
        - /path/to/1688-data-plugin/src/index.js
      env:
        ALIBABA_COOKIE: "${ALIBABA_COOKIE}"
        PROXY_URL: "${PROXY_URL}"
```

### Available Tools

The plugin provides the following MCP tools:

1. **`alibaba_search_products`** - Search 1688 products with keywords and filters
2. **`alibaba_get_product_details`** - Get detailed product information
3. **`alibaba_get_supplier_info`** - Get supplier information and ratings
4. **`alibaba_get_price_history`** - Get price history for a product
5. **`alibaba_check_supplier_rating`** - Check supplier rating metrics
6. **`alibaba_get_minimum_order_quantity`** - Get minimum order quantity (MOQ)
7. **`alibaba_analyze_market_trend`** - Analyze market trends for keywords

### Example Usage via OpenClaw

Once configured, you can use the tools through OpenClaw:

```bash
# Using OpenClaw CLI
openclaw tools call --name alibaba_search_products --arguments '{"keywords": "蓝牙耳机", "filters": {"min_price": 10, "max_price": 100}}'
```

## Project Structure

```
1688-data-plugin/
├── src/
│   ├── index.js              # MCP server entry point
│   ├── auth.js               # Authentication management
│   ├── alibaba-api.js        # 1688 API client
│   ├── tools.js              # MCP tool definitions
│   └── utils.js              # Utility functions
├── config/
│   └── alibaba.yaml.example  # Example configuration
├── examples/
│   ├── basic-usage.js        # Basic usage example
│   └── crawl-test.js         # Comprehensive test script
├── scripts/
│   └── proxy-check.js        # Proxy testing utility
├── test/
│   ├── unit/                 # Unit tests
│   └── integration/          # Integration tests
├── .env.example              # Example environment variables
├── package.json              # Dependencies and scripts
├── mcp.json                  # MCP plugin definition
└── README.md                 # This file
```

## Development

### Building and Testing

```bash
# Install dependencies
npm install

# Run tests
npm test

# Run linting
npm run lint

# Format code
npm run format

# Run examples
npm run crawl-test
```

### Adding New Features

1. **New API endpoints**: Add methods to `src/alibaba-api.js`
2. **New MCP tools**: Add tool definitions to `src/tools.js` and update `mcp.json`
3. **Configuration options**: Update `src/auth.js` and configuration examples

## Best Practices for Web Scraping

1. **Respect robots.txt**: Check `https://www.1688.com/robots.txt` before scraping
2. **Rate limiting**: Use the built-in rate limiter (default: 10 requests/minute)
3. **Random delays**: Enable random jitter between requests
4. **Proxy rotation**: Use proxies for large-scale data collection
5. **User-agent rotation**: The plugin automatically rotates user agents
6. **Error handling**: Implement retry logic for transient failures
7. **Data caching**: Cache responses to avoid redundant requests

## Integration with E-commerce System

This plugin is designed to work with the other e-commerce plugins:

- **Data standardization**: Output normalized data compatible with `ecommerce-db-plugin`
- **Complementary data**: Provides sourcing data to complement `amazon-sp-api-plugin`
- **Unified schema**: Follows the same data schema patterns

Example integration workflow:
1. Search products on 1688 using `alibaba_search_products`
2. Get detailed product information using `alibaba_get_product_details`
3. Check supplier reputation using `alibaba_check_supplier_rating`
4. Store data in ecommerce database using `ecommerce-db-plugin`
5. Compare prices with Amazon using `amazon-sp-api-plugin`

## Troubleshooting

### Common Issues

1. **"Cookie expired or invalid"**
   - Regenerate the cookie by logging into 1688.com
   - Update the `ALIBABA_COOKIE` environment variable

2. **"Request blocked" or CAPTCHA detected**
   - Increase delays between requests
   - Use a proxy server
   - Try the Playwright mode (`USE_PLAYWRIGHT=true`)

3. **"Rate limited"**
   - Decrease request frequency
   - Implement proxy rotation
   - Respect 1688's terms of service

4. **"Playwright not working"**
   - Install Playwright dependencies: `npx playwright install`
   - Ensure Chrome/Chromium is available
   - Check system dependencies

### Debug Mode

Enable debug logging:

```bash
LOG_LEVEL=debug node examples/basic-usage.js
```

## License

MIT

## Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass
5. Submit a pull request

## Support

- Issues: [GitHub Issues](https://github.com/aoyasong/1688-data-plugin/issues)
- Documentation: [OpenClaw Docs](https://docs.openclaw.ai)

## Disclaimer

This plugin is for educational and legitimate business purposes only. Users are responsible for:
- Complying with 1688.com's terms of service
- Respecting intellectual property rights
- Not overloading 1688's servers
- Using data in accordance with applicable laws and regulations

The authors are not responsible for any misuse of this software.

tools